AN EXAMINATION OF THE EFFECTS OF

PARTICIPATION IN HIGH-IMPACT PRACTICES USING

PROPENSITY SCORE ANALYSIS WITH STRUCTURAL EQUATION MODELING

A dissertation submitted to the

Kent State University College
of Education, Health, and Human Services
in partial fulfillment of the requirements
for the degree of Doctor of Philosophy

By

Joanna L. Dickert

December 2021
© Copyright, 2021 by Joanna L. Dickert
All Rights Reserved

ii
 A dissertation written by

Joanna L. Dickert

B.S.B.A. and B.A., Duquesne University, 1998

M.S.Ed., Duquesne University, 2001

Ph.D., Kent State University, 2021

Approved by

______________________________________, Director, Doctoral Dissertation Committee

Jian Li

______________________________________, Member, Doctoral Dissertation Committee

Mark Kretovics

______________________________________, Member, Doctoral Dissertation Committee

Jason Schenker

Accepted by

______________________________________, Director, School of Foundations, Leadership

Kimberly S. Schimmel and Administration

______________________________________, Dean, College of Education, Health and Human

James C. Hannon Services

iii
DICKERT, JOANNA L. Ph.D., December 2021 Research, Measurement, and Statistics

AN EXAMINATION OF THE EFFECTS OF PARTICIPATION IN HIGH-IMPACT

PRACTICES USING PROPENSITY SCORE ANALYSIS WITH STRUCTURAL EQUATION
MODELING (169 pp.)

Director of Dissertation: Jian Li, Ph.D.

This study examined the relationship between cumulative participation in high-impact

practices (HIP) and post-graduation college outcomes using the Educational Longitudinal Study

of 2002 (ELS) dataset. The methodological approach proposed and tested by Leite et al., 2019

incorporated the ability to account for self-selection using propensity score (PS) analysis with a

multiple-group structural equation model (SEM) design, thereby allowing examination of

differences between students who participated in two or more HIPs and those who did not.

Results offered evidence of benefit to perceived importance of postsecondary education in

preparation for adult life from participation in two or more HIP experiences as a main effect.

Additionally, positive and statistically significant differences in perceived importance of

postsecondary education in preparation for adult life were identified for female students, students

from low SES backgrounds, and students who are members of minoritized racial/ethnic

populations. Conversely, no statistically significant main effect in perceived learning and

challenge in professional experiences was identified. Similarly, there were no statistically

significant differences in perceived learning and challenge in professional experiences identified

across groups who participated in two or more HIP experiences and those who did not.
 ACKNOWLEDGMENTS

I am grateful for the support and encouragement of many individuals who have been so

generous with their time and expertise throughout this endeavor. First and foremost, my

dissertation director, Dr. Jian Li, has been a constant source of support and encouragement

throughout these past two years. Although neither of us imagined that the bulk of this work be

undertaken during a global pandemic when you agreed to serve as my director, I am thankful

everyday that you did. You have unfailingly shared your time and expertise and this research

would not have been possible without your advisement. I appreciate all of your efforts to support

my development as a researcher and scholar.

I am likewise thankful for the knowledgeable scholars who have supported me

throughout this journey. Dr. Jason Schenker and Dr. Mark Kretovics have been thoughtful and

dedicated dissertation committee members and I appreciate the feedback that they have shared to

shape my thinking around the research questions guiding the present study, the methods that

allowed me to uncover evidence to answer those questions, and the implications of what the

findings entail for both postsecondary researchers and practitioners. I am particularly grateful to

Dr. Walter Leite whose methodological work was employed in my study. Dr. Leite has likewise

been generous with his time and expertise and I am particularly appreciative of his willingness to

share data files from his own study to help guide my decision making in my own study design.

Even before I embarked on the present study, I have benefitted from the opportunity to

learn from many talented teachers and mentors. In addition to my committee members, my

coursework with Dr. Aryn Karpinski, Dr. Tricia Niesz, and Dr. Michele Donnelly has shaped my

thinking about measurement methods and study design. I have also been fortunate to know many

iv
talented mentors who have challenged me to think critically about postsecondary research, the

methods that are most frequently employed, and strategies for elevating the discourse for

researchers and practitioners. Dr. Holly Hippensteel, Dr. Marsha Lovett, and Dr. Sharon Carver,

I am grateful to each of you for helping me achieve this milestone and I look forward to our

continued collaborations.

Finally, this work would have never come to fruition without the encouragement and

support of my family, especially my three children, who have served as my most enthusiastic

cheerleaders, particularly as we spent much time living and learning at home together during

these pandemic times. Sierra, Evan, and Nathan, I hope that the experience of seeing your mom

tackle a lifelong dream and persist in the face of anticipated (and unanticipated) challenges

inspires you to pursue endeavors that are similarly meaningful to you. While these journeys are

most certainly not easy, they are inevitably worthwhile.

v
 TABLE OF CONTENTS

Page
ACKNOWLEDGMENTS................................................................................................ iv

LIST OF TABLES .........................................................................................................viii

CHAPTER
I. INTRODUCTION ..................................................................................................1
College Outcomes and Student Success .................................................................1
National Study of Student Engagement ..................................................................3
Extending the HIP Discourse ..................................................................................5
Gaps in the HIP Scholarship ...................................................................................7
Research Questions .................................................................................................7

II. REVIEW OF THE LITERATURE .......................................................................10

Foundational HIP Scholarship ..............................................................................11
Emerging Themes and Subsequent HIP Scholarship ............................................15
Continued Examination in the NSSE Discourse ...................................................18
Inquiry Beyond the NSSE Dataset ........................................................................21
Self-Selection into HIP Experiences .....................................................................28

III. METHODOLOGY................................................................................................33
Data Source ...........................................................................................................33
Measures ...............................................................................................................44
Weights .................................................................................................................56
Data Analysis ........................................................................................................57
Conceptual Models................................................................................................64

IV. MANUSCRIPT I...................................................................................................66

Abstract .................................................................................................................66
Introduction ...........................................................................................................66
Theoretical Framework .........................................................................................68
Literature on HIPs and College Outcomes............................................................70
Data Source ...........................................................................................................75
Analytic Sample ....................................................................................................77
Outcome Measure .................................................................................................79
Treatment Measure................................................................................................80
Model Covariates ..................................................................................................81
Weights .................................................................................................................85
Methods .................................................................................................................85
Results ...................................................................................................................89
Analytical Limitations...........................................................................................96
Discussion .............................................................................................................97

vi
V. MANUSCRIPT II ................................................................................................102
Abstract ................................................................................................................102
Introduction ..........................................................................................................102
Theoretical Framework ........................................................................................105
Literature on HIPs and College Outcomes ..........................................................106
Data Source ..........................................................................................................112
Analytic Sample ...................................................................................................114
Outcome Measure ................................................................................................116
Treatment Measure ..............................................................................................117
Model Covariates .................................................................................................118
Weights ................................................................................................................121
Methods................................................................................................................122
Results ..................................................................................................................126
Analytical Limitations .........................................................................................133
Discussion ............................................................................................................134

VI. CONCLUSION ....................................................................................................140

APPENDICES .....................................................................................................143
APPENDIX A: NSSE SCALE OF EDUCATIONALLY PURPOSEFUL
ACTIVITIES..................................................................................................144
APPENDIX B: LIST OF MODEL COVARIATES ............................................146
APPENDIX C: CLASSIFICATION OF INSTRUCTIONAL PROGRAMS (CIP)
CODE AGGREGATION...............................................................................148
APPENDIX D: CONCEPTUAL MODEL FOR PS ESTIMATION ..................150
APPENDIX E: CONCEPTUAL MODEL FOR MANUSCRIPT I .....................152
APPENDIX F: CONCEPTUAL MODEL FOR MANUSCRIPT II....................154

REFERENCES ................................................................................................................156

vii
 LIST OF TABLES

Table

1. Manuscript I: Model Fit Statistics and Indices ..................................................... 90

2. Manuscript I: Standardized Mean Differences (SMD) for Covariate Balance

Evaluation ............................................................................................................. 91

3. Manuscript I: Interaction Estimates from Multiple-Group SEM .......................... 93

4. Manuscript I: P-Values from Sensitivity Analysis ............................................... 96

5. Manuscript II: Model Fit Statistics and Indices ...................................................127

6. Manuscript II: Standardized Mean Differences (SMD) for Covariate Balance

Evaluation ............................................................................................................128

7. Manuscript II: Interaction Estimates from Multiple-Group SEM .......................130

8. Manuscript II: P-Values from Sensitivity Analysis .............................................133

viii
 1

CHAPTER I

INTRODUCTION

Since 2007, the scholarly discourse on college outcomes has foregrounded a set of

undergraduate student experiences that have been termed high-impact practices (HIPs) (Kuh,

2008). Propelled to prominence by the Association of American Colleges and Universities

(AAC&U) via the Liberal Education and America’s Promise initiative (2007), this designation

denotes a set of undergraduate educational experiences that share a set of common attributes

including sustained independent inquiry, frequent interaction with faculty and peers, and regular

feedback, synthesis, and perspective taking (Kuh, 2008). As designated by AAC&U (2007), HIP

experiences include first-year seminars and experiences, common intellectual experiences (i.e.,

core curriculum), learning communities, writing-intensive courses, collaborative assignments

and projects, undergraduate research, diversity and global learning, service learning and

community-based learning, internships, and capstone courses and projects.

College Outcomes and Student Success

The HIP discourse has drawn heavily on traditional models of college outcomes and

student success (Astin, 1984; Astin, 1993; Tinto, 1993; Terenzini et al., 1996). With an emphasis

on the choices that students make regarding how to direct their time and energy during their

respective undergraduate enrollments (Astin, 1984), the interplay between precollege

characteristics, postsecondary experience, and postgraduation outcomes has long been

acknowledged. Astin’s Inputs-Environment-Outputs (I-E-O) model (1993) conceptualized the

inputs as student characteristics at time of entry; the environment as the people, programs, and

experiences with which the student engages as an undergraduate; and the outputs as the student
 2

characteristics at time of departure. Tinto’s Student Departure Model (1993) offered a similar

conceptualization of this synergy. In this model, pre-entry attributes, intended and actualized

student goals and commitments, and academic and social experiences while enrolled served as a

framework for student retention. Terenzini and colleagues (1996) offered further distillation of

these constructs, noting that six primary elements are central to understanding college outcomes:

precollegiate traits, curricular patterns, in-class experiences, out-of-class experiences,

institutional context, and learning outcomes.

More recently, Reason (2009) articulated a more expansive view of college outcomes

with a particular focus on student persistence. Drawing on a prior collaboration (Terenzini &

Reason, 2005), Reason’s framework (2009) foregrounded the importance of precollege

characteristics and experiences that include sociodemographic characteristics as well as family

support albeit with the acknowledgement that the family support dimension represents an aspect

of the literature that has historically been underexplored. Like Tinto (1993), Reason (2009)

separated student dispositions such goals and motivation and academic preparation in high

school from precollegiate characteristics. Additionally, the organizational context of the

postsecondary institution including structural-demographic characteristics such as institutional

control,1 size, and selectivity was foregrounded with a particular emphasis on first institution

attended consistent with the findings of Pascarella and Terenzini (2005). However, Reason

(2009) cautioned against limiting an examination of institutional context to structural-

demographic characteristics, noting that organizational behavior dimensions including

1
Institutional control is an institutional characteristic defined by the Integrated Postsecondary Education Data
System. The three options for a variable that specifies institutional control are public, private not-for-profit, and
private for-profit.
 3

institutional priorities, campus policies, and decision-making structures are equally important, a

consideration affirmed by subsequent scholarship (Ro et al., 2013). Finally, Reason (2009)

offered important distinctions across student experiences, separating the peer environment

including the racial and academic climate from individual student experiences which can occur

in the classroom; in the curriculum via student major and participation in HIPs; and outside the

classroom via participation in programs and activities such as athletics and student organizations.

National Study of Student Engagement (NSSE)

Seminal scholarship regarding the relationship between HIP participation and colleges

outcomes has historically been dominated by studies that use data from the National Study of

Student Engagement (NSSE).2 Using data from the 18 institutions that administered the NSSE at

least once between 2000 and 2003, the foundational study (Kuh, Kinzie, Cruce, et al., 2007) in

this discourse identified multiple positive and statistically significant albeit relatively small

effects for participation in educationally purposeful activities3 for both first-year and graduating

seniors. Participation in educationally purposeful activities during the first year was found to

have a positive relationship with both first-year grade point average (GPA) and persistence to the

second year while a similarly small but significant relationship was identified between

participation in educationally purposeful activities and fourth-year GPA. In addition to these

2
The NSSE is a multi-institutional study conducted annually since 2000 (National Study of Student Engagement,
2001). It is a joint venture of the Indiana University Center for Postsecondary Research, the Indiana University
Center for Survey Research, and the National Center for Higher Education Management Systems. Administered to
both first-year and fourth-year undergraduate students, it is a cross-sectional study that facilitates exploration of self-
reported student experiences and subsequent outcomes at both the onset and conclusion of their respective
undergraduate enrollments.
3
The NSSE scale of educationally purposeful activities used by Kuh, Kinzie, Cruce, et al. (2007) included 19 items
that were designed to measure student engagement. Please note that while a subset of these items directly related to
participation in HIP experiences (e.g., participate in a community-based project as part of a regular course), the
majority of behaviors on the list did not. The full list of items is shown in Appendix A. Coefficient Alpha was .818
for the first-year cohort and .836 for the graduating senior cohort.
 4

main effects, participation in educationally purposeful activities yielded evidence of conditional

effects within the first-year student cohort with positive effects related to first-year GPA for

students who entered with lower ACT scores and positive effects related to persistence to the

second year for African American students.

Although this foundational study focused on the set educationally purposeful activities

which included some HIPs but was not exclusively focused on HIPs, it gained scholarly

prominence via a series of subsequent papers (Kinzie et al., 2008; Kuh, Kinzie, Buckley, et al.,

2007; Kuh et al., 2008) and was also featured in a 2008 AAC&U monograph entitled High

Impact Practices: Who They Are, Who Has Access to The, and Why They Matter (Kuh, 2008).

This monograph also incorporated findings from the 2007 NSSE dataset that specifically

examined the relationship between participation in individual HIP experiences and self-reported

student learning. Results of this analysis were mixed with findings suggesting that participation

in learning communities and service learning yielded the most substantive gains in deep

learning4 for the first-year cohort while an effect of similar magnitude was identified for

participation in undergraduate research for the senior year cohort. The relationships between

participation in other HIPs and deep learning were generally positive and more moderate in

magnitude.

With early findings signaling benefits of HIP participation, Kuh (2008) recommended

that undergraduate students participate in at least two HIP experiences with one situated early in

their enrollment and one sequenced closer to graduation. Using NSSE data collected from 2006-

2008, Finley and McNair (2013) identified a positive relationship between increased

4
As measured by the National Study of Student Engagement in 2007, deep learning practices included a range of
cognitive skills focused on analysis, synthesis, and integration (Kuh, 2008; Laird et al., 2008)
 5

participation in HIP experiences and self-reported gains in general education, practical

competence, and personal and social development. Although these patterns were reflected

generally with the full analytic sample as well as across racial groups, gains appeared to be

particularly salient for first-generation students with self-reported gains in each of the

aforementioned areas exceeding those of continuing generation students (Finley & McNair,

2013).

Extending the HIP Discourse

Despite the positive, albeit modest, effects of participation in HIP experiences identified

in early studies, the literature was not without limitation related to sample selection,

methodology, and outcomes (Brownell & Swaner, 2010). Moreover, as researchers sought to

address some of these limitations, the relationship between participation in HIP experiences and

college outcomes became more nuanced. Even more recent studies that used the NSSE dataset

suggested that learning gains varied by HIP experience (Valentine et al., 2021). Additionally,

these gains varied not just by type of HIP experience but also in magnitude across racial groups

(Valentine et al., 2021; Zilvinskis, 2019). A similar finding was identified related to post-

graduation plans with differential effects identified across HIP experiences (Miller et al., 2018).

As the literature base expanded to include studies with multi-institutional datasets other

than the NSSE, further evidence emerged that clarified and sometimes contradicted the

prevailing discourse. Both the Wabash National Study of Liberal Arts Education (WNS) and the

Educational Longitudinal Study (ELS) offered researchers variables from multiple waves of data

collection, thereby permitting a broader range of pre-college characteristics to be incorporated

into analytic models and more diverse sets of college outcomes to be considered. Studies using
 6

the WNS contributed to the growing body of evidence that learning gains varied across

individual HIP experiences (Kilgo et al., 2015; Seifert et al., 2014) and that an overemphasis on

general effects potentially masked interaction effects for minoritized student populations (Seifert

et al., 2014). Little consensus emerged regarding the relationship between HIP participation and

college completion with some evidence suggesting no relationship existed (Johnson & Stage,

2018) and other findings suggesting not only a positive effect but also a cumulative one

(McDaniel & VanJura, 2020). Finally, post-graduation outcomes began to be examined with

positive and cumulative effects of HIP participation identified for civic engagement in early

adulthood (Myers et al., 2018) and varying effects by major for early career outcomes such as

self-reported earnings and attitudes toward current jobs (Wolniak & Engberg, 2019).

Despite the expansion of the literature base relating to HIP participation and college

outcomes, few studies have specifically accounted for the issue of student choice (Astin, 1984), a

notable omission as HIP experiences are not universally required for graduation. Observational

studies that do not account for the opt-in nature of these activities are subject to issues of

selection bias (Rosenbaum, 2002) in that students who elect to participate may be different from

those who choose not to participate. Propensity score (PS) analysis offers researchers a variety of

techniques including matching, weighting, and stratification to reduce bias (Austin, 2011; Stuart,

2010; Thoemmes & Kim, 2011). Indeed, the studies that have incorporated these types of PS

adjustments in the examination of the relationship between HIP participation and college

outcomes (Bowman & Holmes, 2018; Culver & Bowman, 2019; Provenchar & Kessel, 2019)

have generally found that the absence of PS adjustments resulted in overestimation of the effects

of HIP participation.
 7

Gaps in the HIP Scholarship

While considerable progress has been made to address some of the limitations in the HIP

scholarship (Brownell & Swaner, 2010), further inquiry is warranted. The dominance of

observational studies that have not accounted for selection bias via PS analytic techniques or

other methods in both foundational and contemporary scholarship is particularly notable (Finley

& McNair, 2013; Kilgo et al., 2014; Kuh, Kinzie, Cruce, et al., 2007; Kuh, 2008; McDaniel &

Van Jura, 2020; Miller et al., 2018; Myers et al., 2018; Seifert et al., 2014; Valentine et al., 2021;

Wolniak & Engberg, 2019). Moreover, the studies that have incorporated these types of controls

likewise exhibited some of the limitations identified by Brownell and Swaner (2010) in that they

used data from a single institution (Provenchar & Kassel, 2019) or focused on a single HIP

experience (Bowman & Holmes, 2018; Culver & Holmes, 2019). Additionally, few studies have

identified benefits of cumulative participation in HIP experiences (Finley & McNair, 2013;

McDaniel & Van Jura, 2020; Myers et al., 2018) and of those, just one has focused on post-

graduation outcomes (Myers et al., 2018).

Research Questions

The present study was designed to extend existing scholarship on the relationship

between participation in HIP experiences and college outcomes using the ELS dataset. It

employed PS analysis with structural equation modeling (SEM) using MPlus software to better

understand the relationship between participation in two or more HIP learning experiences as

recommended by Kuh (2008) and post-graduation college outcomes. The study used the same

methodology (Leite et al., 2019) to explore two distinct sets of research questions.
 8

Manuscript I focused on the relationship between cumulative participation in HIP

experiences and perceptions of preparedness for adult life. The research questions guiding

Manuscript I were as follows:

1. Does participation in two or more HIP experiences affect perceived importance of

postsecondary experience in preparation for adult life when controlling for student

precollege sociodemographic characteristics, high school academic preparation,

organizational factors, and individual student experiences?

2. Is the relationship between participation in two or more HIP experiences and

perceived importance of postsecondary experience in preparation for adult life

moderated by student race/ethnicity, student sex, student SES, and high school

academic achievement?

Manuscript II similarly targeted post-graduation outcomes, albeit with a focus on the

relationship between cumulative participation in HIP experiences and perceptions of learning

and challenge in professional experiences. The research questions guiding Manuscript II were as

follows:

1. Does participation in two or more HIP experiences affect perceptions of learning

and challenge in professional experiences when controlling for student precollege

sociodemographic characteristics, high school academic preparation,

organizational factors, and curricular and cocurricular individual student

experiences?

2. Is the relationship between participation in two or more HIP experiences and

perceptions of learning and challenge in professional experiences moderated by

 9

student race/ethnicity, student sex, student SES, and high school academic

achievement?
 10

CHAPTER II

REVIEW OF THE LITERATURE

Over the last 15 years, the nature of the relationship between HIPs and college outcomes

has been explored by multiple researchers using diverse institutional and multi-institutional

datasets. Even as initial research acknowledged that there are limits to what postsecondary

institutions can do to improve college outcomes for all students, particularly those who come to

these institutions with years of educational disadvantage (Kuh, 2008), colleges and universities

have latitude to implement policies and programs to support student engagement that can help to

close these gaps.

However, since 2008, postsecondary institutions have experienced considerable shifts in

both enrollment and financial support structures. The first of these impacts was experienced

following the Great Recession (Barr & Turner, 2013), a historical period that closely followed

the rise to prominence of the earliest HIP scholarship. More proximately, the impacts of the

COVID-19 pandemic are now just beginning to be experienced by postsecondary institutions

(Friga, 2021) and much remains to be understood about longer term implications.

Thus, it is critical that nuances and limitations in the conclusions that can be drawn from

the literature regarding HIP participation be fully examined and understood. This critical

examination is intended to support the ability of postsecondary institutions to deploy limited

financial resources efficiently and effectively in a manner that maximizes the transformative

potential of student engagement. A chronological examination of the HIP scholarship offers

valuable insight into how datasets, models, and methods have evolved over time, resulting in
 11

increased complexity in terms of research design, data analysis, and actionable insights afforded

by the results.

Foundational HIP Scholarship

The seminal scholarship pertaining to HIPs as a set of experiences dates to 2007 with a

study that used data from the NSSE (Kuh, Kinzie, Cruce, et al., 2007). Sponsored by the Lumina

Foundation, this initial study focused on a set of 18 postsecondary institutions who administered

the NSSE at least once between 2000 and 2003. This study measured student engagement via the

NSSE scale of educationally purposeful activities, a set of experiences that included but was not

exclusive to HIPs as shown in Appendix A.

First-Year Outcomes

Using Ordinary Least Squares (OLS) regression, Kuh, Kinzie, Cruce, et al. (2007)

identified a positive relationship between student engagement in educationally purposeful

activities and first-year GPA for the first-year student cohort (n = 6,000). Sociodemographic

control variables for this study included race/ethnicity, number of parents with a 4-year degree,

and parent income. High school experiences including number of honors courses taken in high

school and number of high school extracurricular activities were accounted for in the model

along with high school grade point average (GPA) and ACT score which served as indicators of

prior academic achievement.

Kuh, Kinzie, Cruce, et al. (2007) found that for every one standard deviation increase in

student engagement in educationally purposeful activities, first-year GPA increased by .04 points

(p < .001). They also identified a small positive relationship between student engagement in

educationally purposeful activities and persistence to the second year of college with controls for
 12

student sociodemographic characteristics, high school experiences, prior academic achievement,

and unmet financial need. Participants with a level of engagement in educationally purposeful

activities that was one standard deviation above the mean had 0.91 probability of persisting

versus 0.85 for participants with a level of engagement in educational purposeful activities that

was one standard deviation below the mean.

The findings of Kuh, Kinzie, Cruce, et al. (2007) also signaled the presence of

conditional effects on first-year GPA and persistence to the second year of college at the same

institution. More specifically, the direct effects of participation in educationally purposeful

activities on first-year GPA varied by prior academic achievement. For every one standard

deviation increase in student engagement in educationally purposeful activities, participants who

entered with an ACT score of 20 experienced a 0.06 increase in first-year GPA versus a 0.04

increase for participants who entered with an ACT score of 24 and a 0.02 increase for

participants who entered with an ACT score of 28.

Similarly, the direct effects of participation in educationally purposeful activities on first-

year GPA varied by race/ethnicity albeit solely for White and Hispanic participants. For every

one standard deviation increase in student engagement in educationally purposeful activities,

Hispanic participants experienced a 0.11 increase in first-year GPA versus a 0.03 increase for

White participants. Kuh, Kinzie, Cruce, et al. (2007) identified a similarly constrained

conditional effect related to persistence to the second year of college, noting that African

American participants experienced a larger benefit from increased engagement in educationally

purposeful activities than their White peers in the analytic sample.

Senior Year Outcomes

 13

Also using Ordinary Least Squares (OLS) regression, a positive relationship between

student engagement in educationally purposeful activities and senior-year GPA was identified

for the senior cohort (n = 5,000) (Kuh, Kinzie, Cruce, et al., 2007). Controls for student

sociodemographic characteristics and high school experiences remained the same as for the first-

year cohort. However, for the senior student cohort, prior academic year GPA replaced high

school GPA and ACT score as indicators of prior academic achievement. An increase of one

standard deviation in student engagement corresponded with an increase of .03 points in senior-

year GPA (p < .001). Unlike the first-year cohort, effects did not vary by race/ethnicity, thereby

signaling that the conditional effects present in the first-year cohort may not extend to senior

year.

Thus, the small but statistically significant relationship between participation in

educationally purposeful activities and measures of educational attainment including academic

achievement and persistence to second year identified by Kuh, Kinzie, Cruce, et al. (2007)

served as the foundation for the HIP discourse. Results of this study were referenced in multiple

white papers and peer-reviewed articles (Kinzie et al., 2008; Kuh, Kinzie, Buckley, et al., 2007;

Kuh et al., 2008) and were featured prominently in the High-Impact Practices: Who They Are,

Who Has Access to Them, and Why They Matter monograph published by AAC&U (Kuh, 2008).

Learning Gains

The AAC&U monograph (Kuh, 2008) similarly foregrounded results of the 2007 NSSE

annual report focused on the relationship between HIP participation and student perception of

learning gains (National Study of Student Engagement, 2007) using OLS regression. The

analytic sample for this study included both first-year students (n = 149,181) and senior students
 14

(n = 148,902) drawn from a random sample across 587 institutions. Gain measures included

indicators of deep learning such as higher order thinking, integrative learning, and reflective

learning as well as an aggregate measure of overall deep learning. Three self-reported gains

scales focused on general education, personal and social development, and practical

competencies were also examined. Student-level control variables in the regression model

included gender, race/ethnicity, enrollment status, major, transfer status, parents’ education, and

student grades. Institutional control variables within the model included Carnegie classification,

institutional selectivity, and institutional control.

Participation in learning communities was the sole HIP experience examined for the first-

year cohort (National Study of Student Engagement, 2007). Participation in learning

communities corresponded with gains across all learning measures at p < .001 with varying

effect sizes as measured by unstandardized regression coefficients (B). The strongest effects

associated with participation in learning communities included overall deep learning and

integrative learning (National Study of Student Engagement, 2007) (B > .30, p < .001). All other

effects were more moderate (B > .10, p < .001).

Participation in undergraduate research, study abroad, and senior capstone were the three

HIP experiences under study for the senior cohort. Effects for undergraduate research mirrored

those for learning communities with the strongest effects once again associated with overall deep

learning and integrative learning (National Study of Student Engagement, 2007) (B > .30, p <

.001). Effects for study abroad were more moderate with gains related to overall deep learning,

integrative learning, reflective learning, and personal and social development representing the

most substantive effects (National Study of Student Engagement, 2007) (B > .10, p < .001).
 15

Participation in senior capstone experiences yielded similar moderate effect sizes for all gain

scales (B > .10, p < .001).

Emerging Themes and Subsequent HIP Scholarship

Following the initial empirical support for the relationship between HIP participation and

college outcomes, AAC&U subsequently supported additional inquiry related to HIP

experiences, resulting in the publication of two monographs. Brownell and Swaner (2010)

reviewed existing scholarship related to individual HIP experiences and documented those

findings in an AAC&U monograph entitled Five-High Impact Practices: Research on Learning

Outcomes, Completion, and Quality. Building on the work of Kuh (2008) and Brownell and

Swaner (2010), Finley and McNair (2013) brought a particular focus on underserved student

engagement in HIP experiences, using both quantitative measures to consider effects as well as

qualitative measures to probe potential barriers to participation.

Outcomes for Individual HIP Experiences

Although Kuh, Kinzie, Cruce, et al. (2007) were the first to study the impact of

cumulative participation in HIP experiences, individual HIP experiences have long been

explored by postsecondary researchers, even before the advent of the HIP designation. Brownell

and Swaner (2010) reviewed the literature pertaining to first-year seminars, learning

communities, undergraduate research, service learning, and capstone courses and projects. While

not specifically designated as a meta-analysis, Brownell and Swaner (2010) identified outcomes

related to persistence for all but capstone courses and projects as well as academic achievement

as measured by GPA for first-year seminars, learning communities, and service learning. In

terms of learning gains, the body of scholarship yielded evidence of positive relationships
 16

between self-reported gains in critical thinking with participation in learning communities,

service learning, and undergraduate research.

Despite the evidence of outcomes that were shared across individual practices, Brownell

and Swaner (2010) likewise noted multiple gaps in the literature base including:

• Overemphasis on academic achievement as measured by GPA and graduation

with limited attention to direct measures of student learning

• Lack of attention to the experience of underserved students which mitigates the

ability to understand interaction effects

• Prevalence of studies focused on single institutions with limited ability to

generalize beyond the singular institutional context

• Dominance of observational studies that rely on convenience samples with no

correction for bias associated with self-selection to participate or use of

comparison groups

• Emphasis on short-term impacts of participation with little consideration of

longer-term effects

Disparities in HIP Participation

Despite the evidence suggesting greater gains in areas related to educational attainment

such as academic achievement and first-to-second year retention rates for minoritized students

who have participated in one or more high-impact learning experiences (Kuh, Kinzie, Cruce, et

al., 2007), Finley and McNair (2013) noted that first-generation student as well as African

American and Hispanic students have a greater likelihood of foregoing these high-impact

learning experiences altogether or engaging in fewer experiences than other groups of students.
 17

Using NSSE data5 collected between 2006 and 2008 from 38 public institutions in California,

Oregon, and Wisconsin (n = 25,336), Finley and McNair (2013) examined the participation rates

in HIP experiences as well as the relationship between the number of HIP experiences and

student perceptions of their learning using OLS regression. Their findings indicate that while

sample members reported mean participation in 1.30 HIP experiences, disaggregation of

participation rates yielded evidence of disparities. White students (M = 1.38) engaged in more

HIP experiences than African American students (M = 1.29). While the aforementioned

difference was not statistically significant, statistically significant differences did emerge

between White students and Hispanic students (M = 1.27, p < .05), and Asian American students

(M = 1.22, p < .05). Differences between first-generation students (M = 1.24) and continuing

generation students (M = 1.45) were not statistically significant.

Finley and McNair (2013) also examined cumulative gains associated with participation

in larger numbers of HIP experiences. With scores calculated based on self-reported measures of

deep learning and learning gains standardized on a 100-point scale, self-reported gains in general

education, practical competence, and personal and social development increased with

participation in increasing numbers of HIP experiences. Their findings suggest that first-

generation students reported varying gain levels that exceeded those for continuing generation

students for all measures. Moreover, across all scales, disparities in perceptions of learning were

smallest for students who participated in 5-6 HIP experiences.

5
The NSSE measures for this study consisted of a new scale focused on self-reported participation in six HIP
experiences: learning communities, service-learning courses, study abroad experiences, internships, capstone
courses or culminating senior experience, and undergraduate research. The 4-point response scale consisted of
“Never,” “Sometimes,” “Often,” and “Very Often.” For service learning, the frame of reference for the respondents
to report participation was the current academic year. For all other experiences, the frame of reference for the
respondents to report participation was the duration of the student’s undergraduate enrollment.
 18

The findings of Finley and McNair (2013) were slightly more nuanced when

disaggregated based on respondent racial and ethnic background of sample members. While

precise gain numbers were not reported, Finley and McNair (2013) provided visualizations along

with analysis on the direction and magnitude of the gains. Increased participation in HIP

experiences yielded increased self-perception of deep learning for all students across the racial

and ethnic background categories. Moreover, for Asian American and White students, self-

reported gains in general education, practical competence, and personal and social development

likewise increased with higher participation in HIP experiences. However, for African American

and Hispanic students, self-reported gains in general education, practical competence, and

personal and social development plateaued or declined with participation in 5-6 HIP experiences.

Continued Examination in the NSSE Discourse

Most recently, NSSE data from survey administrations between 2014 and 2020 was used

to explore the relationship between participation in HIPs and self-reported learning gains using a

series of OLS regression analyses (Valentine et al., 2021). The analytic sample for this study

consisted of NSSE participants from 15 four-year institutions in Georgia, Wisconsin, and

Montana (n = 28, 500). This study considered participation in six HIP experiences: learning

communities, service learning, study abroad, undergraduate research, internships, and capstone

experiences. Gain scales included academic learning gains and practical learning gains as well as

a measure of overall learning gains that aggregated the two scales. Student-level control

variables included student race/ethnicity, gender, age, class, first-generation student status, and

transfer status.
 19

Valentine et al. (2021) identified significant positive increases across all three scales for

participation in all HIPs except study abroad. Additionally, participation in service learning was

found to have the largest self-reported increase in both academic and practical learning gains.

However, examination of perceived gains for Black and Hispanic sample members yielded more

nuanced findings. While both Black and Hispanic sample members experienced positive and

significant gains from participation in service learning, Black students perceived learning gains

from participation in internships, which did not extend to study abroad experiences, while

Hispanic students self-identified learning gains from participation in study abroad experiences

but not internships. While both Black and Hispanic sample members perceived positive and

significant academic, practical, and overall learning gains from participation in learning

communities, undergraduate research and capstone experiences, Black students identified

smaller perceived academic learning gains from participation in undergraduate research

compared to Hispanic students and to all students in the analytic sample.

Topical NSSE Datasets

Aside from the standard sets of items included in the NSSE instrument, several topical

modules are available to accompany the main instrument (National Study of Student

Engagement, 2021). These items can be added to institutional administrations of the NSSE to

target specific priority areas. The Senior Transitions Module of the NSSE was designed to gather

data regarding post-graduation plans that are known to student respondents during the spring

semester of their senior year. Measures included self-reported post-graduation plans, perceived

connections between academic major and future plans, and self-reported confidence in skill

development. Miller et al. (2018) used student responses from the 2015 administration of this
 20

module (n  31,000) to explore the relationships between participation in six HIP experiences

and post-graduation college outcomes using multilevel logistic regression. A particular area of

focus in this study was the relationship between participation in HIPs and immediate plans after

graduation (i.e., seeking employment versus enrolling in graduate school). Student control

variables included gender, enrollment status, race/ethnicity, age, first-generation student status,

and college grades. The model also included controls for postsecondary student experiences

including campus residence, fraternity/sorority affiliation, online learning enrollment, academic

major, employment, and graduate degree aspirations. Institutional controls included basic

Carnegie classification, sector, and selectivity.

Results indicated that the relationship varied by type of HIP experience. Seniors who

participated in internships (+25%), capstone experiences (+22%), and study abroad (+12%) had

higher odds of seeking employment (Miller et al., 2018). Sample members who participated in

research (+65%) with a faculty member had higher odds of enrolling in graduate school. No

statistically significant effects on post-graduation plans were identified for participation in a

learning community or service learning.

Experimental NSSE Datasets

In 2015, the Indiana Center for Postsecondary Research piloted an experimental set of

items designed to measure the quality of the HIP experience in undergraduate research,

internships, and capstone experiences. Aiming to extend the scholarly discourse beyond

participation in HIPs, these items were targeted to features of these experiences that enhance

their impact. Features that were measured in the experimental item set included high

expectations, student time, collaboration with peers, faculty feedback, integrative learning, real-
 21

world application, and public demonstration. Zilvinskis (2019) used multilevel modeling to

explore the relationship between student responses on the experimental item set and five

outcome variables for students who participated in undergraduate research (n = 4,287),

internships (n = 2,885), and senior capstone experiences (n = 1,487). These outcomes measures

included standardized NSSE measures related to higher-order learning, effective teaching

practices, supportive environment, and GPA along with a measure of student satisfaction.

Student-level control variables included gender, race/ethnicity, first-generation student status,

transfer status, and age while institution-level control variables included institutional type,

control, undergraduate enrollment, and selectivity.

High expectations, faculty feedback, and real-world application were found to have

significant positive relationships with nearly all the outcomes in the study across the three HIP

experiences. Public demonstration was found to have no relationship with the outcomes across

the three HIP experiences while student time had negative relationships with three outcomes:

effective teaching, supportive environment, and satisfaction. Although few significant findings

were identified relative to interaction effects, it is important to note that HIP quality indicators

demonstrated inconsistent relationships with the outcomes under study across the three HIPs for

Black or African American students and Hispanic or Latino students.

Inquiry Beyond the NSSE Dataset

Although HIPs continued to be explored and foregrounded in NSSE discourse, the

foundational scholarship which was initially centered around the NSSE dataset has been

extended to include other national datasets such as the Wabash National Study of Liberal Arts
 22

Education (WNS) 6 and the Educational Longitudinal Study (ELS)7. The longitudinal designs of

both the WNS and ELS not only provided distinct analytic samples with a diverse set of

variables but also permitted a broader range of research questions including pretest/posttest

measures of student learning as well as post-graduation college outcomes. Additionally, unlike

the NSSE or even the WNS, the ELS was a nationally representative dataset which, with the

appropriate study design, provided opportunities for generalization of the conclusions beyond the

sample under study.

HIP Participation and Student Learning

Continuing the line of inquiry related to conditional effects of HIP participation on

student outcomes identified by Finley and McNair (2013), Seifert et al. (2014) explored the

conditional effects of participation in HIPs while extending the HIP discourse beyond the NSSE

dataset by using data from the WNS. With a longitudinal pretest/posttest design, this study

consisted of a series of OLS regression models testing for general and interaction effects of

participation in HIP/good practices8 on effective reasoning and problem solving as well as

inclination to inquire and lifelong learning during their senior year of undergraduate study.

Effective reasoning and problem solving was measured by the Collegiate Assessment of

6
Funded by the Center of Inquiry in the Liberal Arts at Wabash College, the WNS focused on undergraduate student
learning gains across multiple learning outcomes at 17 four-year colleges and universities in the United States.
Pretest data from incoming first-year undergraduate students was collected in fall 2006 followed by a post-test data
collection in spring 2010. A total of 2,212 student respondents participated in both rounds of data collection.
7
Described in greater detail in Chapter 3, the ELS is a longitudinal study sponsored by the National Center for
Educational Statistics (NCES). The study included four rounds of data collection with 2002 serving as the base year
of the study with subsequent follow-ups in 2004, 2006, and 2012. During the baseline year, participants were
completing tenth grade in their respective high schools. Follow-up studies were sequenced to capture key moments
of transition.
8
Seven dimensions of HIP/good practices served as predictor variables: individual research with a faculty member,
quality of faculty-student interaction, quality of teaching received, academic challenge and high expectations,
cooperative learning experiences, cocurricular involvement and positive interactions with peers, and involvement in
diversity experiences (Seifert et al., 2014).
 23

Academic Proficiency (CAAP) Critical Thinking Test (CTT). Inclination to inquire was

measured by the Need for Cognition Scale (NCS) while lifelong learning was targeted by the

Positive Attitude Toward Literacy (PATL) scale. The regression models included student

sociodemographic characteristics such as sex, race, and parental graduate education as well as

precollege characteristics that included precollege measures of the three dependent variables,

ACT (or SAT equivalent) score, and measures of student motivation. Additional control

variables included institutional context (i.e., whether the student was enrolled at a liberal arts

institution) and student’s major field of study.

While significant positive general effects across all outcome measures for at least one of

the HIP/good practices examined were identified, subsequent examination of the interaction

effects led Seifert et al. (2014) to caution that these general effects were subject to

misinterpretation without accounting for interaction effects. For example, when considering

experiences with diverse others, general effects were identified for the CAAP critical thinking

measure and for the attitudinal PATL measure. However, subsequent analysis revealed that

compensatory effects of participation were found for students with lower precollege scores on

the CAAP and PATL measures. That is, students who entered with the lowest scores on the

pretest outcome measures realized the largest gains from participation in the HIPs under study.

These interaction effects even extended to nonsignificant general effects. For example, while the

general effect of participation in undergraduate research on CAAP critical thinking measures was

found to be nonsignificant, a significant interaction was identified for students who entered their

undergraduate studies with the lowest scores on the pretest CAAP critical thinking measures.
 24

Building on the work of Seifert et al. (2014), Kilgo et al. (2015) extended the inquiry to

include the effects of participation in the ten HIPs identified by Kuh (2008) using the WNS

dataset in a similar longitudinal pretest/posttest design and OLS regression analysis. The

dependent variables of the interest were likewise expanded to include moral reasoning,

intercultural effectiveness, and socially responsible leadership. Corresponding measures

including the Defining Issues Test (DIT-2) for moral reasoning, the Miville-Guzman

Universality-Diversity Scale (M-GUDS), the Openness to Diversity /Challenge (ODC) scale, and

the Socially Responsible Leadership Scale (SLRS) were added to the CAAP CTT, NCS, and

PATL as outcome measures. The regression models used by Kilgo et al. (2015) included the

same student demographic variables as the prior analysis (Seifert et al., 2014). Precollege

measures on all outcome measures were likewise incorporated as well as ACT (or SAT

equivalent) score, motivation, and high school involvement in volunteerism, employment, and

extracurricular activities. Controls for institutional context were expanded to include institutional

type (i.e., research university, regional university, or liberal arts college) while weekly work

hours and fraternity/sorority membership were added to the model along with the student’s major

field of study.

Of the 10 HIPs examined, only undergraduate research and active and collaborative

learning were found to have significant positive effects across most of the aforementioned

outcome measures with effect sizes ranging from .06 to .21 of a standard deviation. A

combination of significant positive and negative effects across the outcome measures was

identified for participation in study abroad, internships, service learning, and capstone

courses/experiences while no positive effects across the outcome measures were found for
 25

participation in first-year seminars, academic learning communities, and writing-intensive

courses.

HIP Participation and College Completion

Taking a different approach than prior studies that used datasets with student-level data,

Johnson and Stage (2018) used institution-level data to explore the relationship between the 10

HIP practices identified by Kuh (2008) and institutional graduation rates at four-year public

colleges and universities. This study focused on four-year public institutions granting

baccalaureate degrees that enrolled 10,000 or more undergraduates (n = 101). Controlling for

institutional selectivity and Carnegie Classification, Johnson and Stage (2018) used multiple

regression to explore the relationship between participation in HIPs and graduation rates. They

found that participation in eight of the 10 HIPs had no significant relationship with four-year or

six-year graduation rates. Participation in research with a faculty member at the least-selective

institutions had a statistically significant and high positive correlation with six-year graduation

rate. Participation in internships had a slightly negative relationship with four-year graduation

rates but no relationship with six-year graduation rates. Moreover, participation in first-year

seminars was found to have slightly negative correlations with both four-year and six-year

graduation rates.

A study conducted by McDaniel and Van Jura (2020) likewise examined the relationship

between HIP participation and college completion. In contrast to Johnson and Stage’s approach

(2018), this study used student-level data from the ELS. The longitudinal nature of this national

dataset allowed the researchers to explore the relationship between participation in four HIPs

(internships, research with a faculty member, study abroad, and service learning) and likelihood
 26

of college completion within six years (n = 4,548). Using logistic regression as the primary

analytic method, McDaniel and Van Jura (2020) controlled for a variety of sociodemographic

variables including sex, race/ethnicity, first-generation student status, and socioeconomic status.

Precollege experiences such as academic achievement in high school and cocurricular

involvement in high school were also incorporated into the model as was postsecondary

institution type.

Net of student demographics and experiences, the likelihood of completing a bachelor’s

degree was positively and significantly influenced by participation in each of the HIPs under

study. Moreover, the effect of participation in HIPs was found to be cumulative with the odds of

college completion increasing by 1.81 for participation in each additional HIP. However, despite

the significance of the general effects, no interaction effects were found for gender, race, and

first-generation status.

HIP Participation and Post-Graduation Outcomes

As previously noted, the presence of a third wave of data collection afforded researchers

who opted to use the ELS dataset the opportunity to consider post-graduation college outcomes

for sample members who had completed undergraduate degrees prior to the final wave of data

collection, thereby opening up new possibilities for inquiry that were not available to researchers

using the NSSE or WNS datasets. Myers et al. (2018) used the ELS dataset to explore the

relationship between participation in HIPs and participation in civic engagement in adulthood

using a series of binomial regression models (n = 6,440). Arguably one of the most complex sets

of models in the HIP discourse, these models included a composite outcome variable, several

treatment variables, and a comprehensive range of diverse covariates. The outcome variable in
 27

the models consisted of a composite variable noting cumulative participation in unpaid

community services from 2010-2012. Five HIP experience variables were considered in the

model: internships, research with a faculty member, study abroad, community-based project, and

capstone experiences. The model accounted for student sociodemographic characteristics such as

sex, race/ethnicity, and SES. Pre-college academic experiences such as high school type, high

school extracurricular involvement, and civic engagement and orientation in high school were

included along with high school academic outcomes, high school GPA, and standardized test

scores. Institution-level control variables included institutional control and selectivity. First

major and civic engagement as an undergraduate served as curricular and cocurricular student

experience covariates. Finally, outcome covariates included current postsecondary enrollment,

postsecondary degree attainment, current income, marital status, parental status, living

arrangements, and general civic orientation.

When all covariates were incorporated into the model, positive and significant

relationships were identified between civic engagement in adulthood and three of the five HIP

experiences under study, namely internships, undergraduate research, and community-based

projects. Moreover, benefits of cumulative participation in HIPs were identified with

participation in multiple HIPs yielding increased levels of civic engagement in adulthood,

thereby providing empirical support for the benefits of cumulative participation initially

promoted by Kuh (2008) and previously examined by Finley and McNair (2013).

Wolniak and Engberg (2019) also used the ELS dataset to study the relationship between

participation in internships, research with a faculty member, study abroad, community-based

projects, and capstone experiences and early career outcomes such as self-reported earnings and
 28

attitudes toward their current jobs. The OLS regression models for this study also included

major, relationship between major and current job, and graduate degree attainment as

independent variables. Sociodemographic control variables included race, sex, and SES while

standardized test scores served as indicators of pre-college academic achievement. Institutional

control and selectivity served as the institution-level control variables. Similar to Myers et al.

(2018), post-college experience covariates such as marital status, number of dependents, and

region of employment were likewise incorporated in the model.

Wolniak and Engberg (2019) limited the analytic sample for this study (n = 2,860) to

ELS respondents who completed at least a bachelor’s degree and were full-time employees at the

time of the third follow-up survey who earned at least the federal minimum wage and worked for

at least 26 weeks. For this sample, no consistent general effects of participation in HIP

experiences were found. Rather, the relationship between participation across the set of HIP

experiences and early career outcomes not only varied but was not uniformly positive. However,

interaction effects based on college major were identified with participation in different HIP

experiences yielding different post-graduation outcomes for different majors.

Self-Selection into HIP Experiences

Despite the depth and diversity of the scholarship surrounding participation in HIP

experiences, many of these studies were observational and relatively few had specifically

accounted for the issue of selection bias in study design. Since postsecondary institutions do not

uniformly require them for graduation, many undergraduate students need to elect to participate

in HIPs. This feature of HIP experiences increases the possibility that students who opt in might

differ from student who do not elect to participate (Rosenbaum, 2002). Moreover, the studies
 29

that attempted to account for selection bias have often focused on a single institution or single

type of HIP experience, both of which are limitations within the body of HIP scholarship as

noted by Brownell and Swaner (2010).

Provencher and Kassel (2019) used propensity score matching (PSM) to study the

relationship between cumulative HIP participation during the second year of enrollment and

subsequent retention to the third year of enrollment. Using data from first-time full-time

undergraduate students enrolled at a private liberal arts college (n = 510), the study focused on

participation in any of the following five HIPs: common intellectual experiences, community-

engaged courses, diversity learning experiences, internships, and undergraduate research. All

covariates in the model were student-level and many of the common sociodemographic variables

such as gender, race, first-generation status, and Pell grant receipt status were included. High

school GPA served as a covariate indicating high school academic achievement while curricular

and cocurricular experience covariates included HIP participation during the first year of

enrollment, athletic award receipt, school of major, commuter status, and cumulative first-year

GPA.

After PSM was applied, Provenchar and Kessel (2019) found that the average treatment

effect on students who participated in a HIP experience during sophomore year on retention was

12 percentage points. Moreover, Provencher and Kassel (2019) noted that the exclusion of PSM

from the analysis resulted in overestimation of this effect for participants. Evaluation of alternate

models which included HIP participation in the first year and HIP participation in the first and

second year combined were not significant, thereby suggesting that for this sample, HIP
 30

participation in the sophomore year was a stronger predictor of retention to third year of

enrollment.

The multi-institutional WNS dataset has also been employed to consider questions related

to selection bias. Bowman and Holmes (2018) used PS analyses to better understand the

relationship between participation in undergraduate research in the first year and a broad set of

student outcomes including first-year GPA, cumulative GPA over four years, university

satisfaction, graduate degree aspirations, university retention across all four years, and on-time

graduation. Multiple sets of student and institutional level covariates were used for the PS

analysis to reduce selection bias (Steiner et al., 2015). Student sociodemographic variables

included race/ethnicity, sex, and parent education. In addition to standardized test scores and

high school GPA, additional pre-college characteristics included academic motivation, need for

cognition, personal goals related to professional success, contributing to science, and political

and social involvement as well as self-reported measures of pre-college academic and social

engagement. Major and highest intended degree level were also incorporated as covariates in the

PS model. Finally, institution type (i.e., regional or research) and institutional selectivity were

included as institution-level covariates.

Following the PS analysis which was conducted via stratification due to the nested nature

of the dataset, hierarchical linear modeling (HML) and hierarchical generalized linear modeling

(HGLM) analysis were used to examine the relationship between participation in undergraduate

research and the student success outcomes. Results of these analyses were mixed with

participation in undergraduate research demonstrating a positive and significant relationship with

first-year university satisfaction (B = .176) and fourth-year GPA (B = .165). No other statistically
 31

significant relationships were identified; moreover, negative albeit non-significant relationships

were identified for a subset of the outcomes included graduate degree aspirations and retention to

fall of the fourth year.

Culver and Bowman (2019) adapted the methodology of Bowman and Holmes (2018) in

a study of participation in first-year seminars and college success outcomes. Once again using

the WNS dataset, Culver and Bowman used PS analysis followed by HLM and HGLM to study

the same set of outcome variables as Bowman and Holmes (2018). A similarly broad set of

covariates was incorporated into the PS analysis. In addition to race/ethnicity, sex, and parent

education, status as a native English speaker was incorporated as a student sociodemographic

covariates. Precollege characteristic covariates included academic motivation, need for

cognition, and socially responsible leadership as well as high school GPA and standardized test

scores. Highest intended degree was incorporated as an indicator of post-graduation college

outcomes. Institution-level covariates included institutional requirements for first-year seminar

enrollment and first-year seminar offering category with two categories: (1) academic/inquiry-

based seminar and (2) Orientation/student success seminar.

Results suggest that an absence of PS adjustments in the models results in overestimation

of the effects of participation in first-year seminars. When PS adjustments are included, Culver

and Bowman (2019) identified a positive and significant relationship between first-year seminar

enrollment and a single outcome variable: first-year college satisfaction. This effect was found to

be .08 standard deviations (SD) when using stratification for PS adjustment and .09 SD when

inverse probability of treatment weighting (IPTW) for PS adjustments. Varying non-significant

effects were identified for college satisfaction in the fourth year, GPA in the first or fourth year,
 32

year-to-year retention, and graduation within four years. When considering how effects might

vary by seminar type, few significant effects were identified, and a subset of those effects

disappeared after PS adjustments were made. The sole remaining effect mirrored the main effect

in that participation in academic/inquiry based seminars had a positive and significant effect (.07

SD) on college satisfaction at the end of the first year.

 33

Chapter III

METHODOLOGY

This chapter provides an overview of the methodological approach to answering the

research questions guiding the present study. It presents a summary of the ELS data source

including general data structure, sampling procedures, data collection methods and formats, and

constituent instrumentation. Additionally, the chapter contains descriptions of the treatment

measure, the outcome variables, and the covariates in the model as well as an overview of

distinct steps in the data analysis including missing data treatment, PS analysis, structural SEM,

and sensitivity analysis.

The use of PS analysis with SEM as described and tested by Leite et al., 2019 was

designed to extend the HIP discourse by combining multiple analytic techniques to examine the

relationship between participation in HIP experience and post-graduation college outcomes. This

approach incorporated the ability to account for self-selection using PS analysis as well as the

complex relationships among diverse variables using SEM. With a multiple-group SEM design,

the methodology allows for examination of differences between students who participated in two

or more HIPs and those who did not. Finally, the use of sensitivity analysis permitted

consideration of the degree to which omitted variables may influence the results of model testing

(Harring et al., 2017).

Data Source

Sponsored by the National Center for Educational Statistics (NCES), the Educational

Longitudinal Study (ELS) served as the data source for the present study. It was designed to

examine moments of significant transition in students’ educational experiences beginning in high

 34

school and continuing into postsecondary education as well as their careers. High school

completion, college access and persistence, and postsecondary outcomes were central foci of this

study along with college access and choice. The ELS aggregated information related to factors

that might influence decision making at these points of transition including student background,

family support, high school experiences, postsecondary experiences, and employment

experiences as well as high school, college, and post-college outcomes (Ingels et al., 2004).

The ELS represented the fourth in a series of longitudinal studies administered by NCES.

It followed previous national longitudinal studies including the National Longitudinal Study of

the High School Class of 1972 (NLS-72), the High School and Beyond (HS&B) study in 1980,

and the National Education Longitudinal Study of 1988 (NELS:88). It was followed by the High

School Longitudinal Study of 2009 (HSLS: 2009). The ELS was designed to elicit timely and

relevant responses from participants with updated survey items while also preserving the ability

to draw comparisons with the results of previous longitudinal studies (Ingels et al., 2004).

The ELS dataset was particularly suited to the present study as it contained variables that

correspond with a range of factors associated with both traditional (Astin, 1984; Astin, 1993;

Tinto, 1993; Terenzini et al., 1997) and contemporary college outcomes scholarship (Reason,

2009; Terenzini & Reason, 2005; Ro et al., 2013). Information pertaining to students’ precollege

background including sociodemographic characteristics, academic preparation and performance

in high school, and student dispositions was present in the dataset. Structural-demographic

organizational factors were likewise present as were curricular and cocurricular individual

student experiences. That said, other elements of particular focus in the contemporary college

outcomes discourse (Reason, 2009; Terenzini & Reason, 2005; Ro et al., 2013) including
 35

organizational behavior measures, peer environment indicators such as campus racial climate and

campus academic climate, and individual classroom experiences were not included and remain

topics for future research.

Of particular relevance were the ELS items related to high-impact practice (HIP)

participation. A composite variable representing cumulative student participation in a range of

HIP experience types was present in the ELS dataset. Dichotomous indicators representing

individual participation in internships, research, study abroad, community-based projects, and

capstone experiences were likewise included. While these were indirect measures that relied

solely on self-reported participation, they were important data points that not only reflected

participation in certain types of HIPs but also facilitated understanding of cumulative

participation across different types of experiences.

The HIP participation measures in the ELS were distinct among available NCES datasets.

Collected in 2012, these data were released for public use in 2014. While more recent studies

including the Beginning Postsecondary Students Longitudinal Study (BPSL) of 2012 and the

HSLS: 2009 did include measures of HIP participation, neither was as comprehensive across

diverse experience types as the ELS. The BPSL dataset included questions about participation in

study abroad but not internships, research, community-based projects, or capstone experiences.

The HSLS:2009 dataset included only three HIP activities: research, study abroad, and

community-based projects.

ELS Sampling Procedure

The initial population for the ELS study was comprised of high school students enrolled

in the tenth grade during spring 2002 in the United States (Ingels et al., 2004). The population
 36

included students in regular public schools, State Department of Education schools, and charter

schools as well as those in private schools. Schools with no tenth grade, ungraded schools,

Bureau of Indian Affairs (BIA) schools, special education schools, vocational schools with no

direct student enrollment, schools in detention centers or correctional facilities, and Department

of Defense schools located outside of the United States were excluded.

The initial sample for the ELS was selected using a stratified random sampling approach

that consisted of two stages of selection (Ingels et al., 2004). The first stage of random sampling

occurred at the high school level with a target of approximately 800 schools to be selected for

participation. The study design included 600 public schools and 200 private schools. Further

stratifications were made by United States Census division for public schools and by United

States Census region for private schools. Anticipated nonresponse based on historical response

rates in prior longitudinal studies such as the NELS: 88 and HS&B necessitated a sample of

1,263 schools with 752 ultimately agreeing to participate in the study (Ingels et al., 2004). Of

these 752 schools, 580 were public schools and 172 were private schools.

In the second stage of sampling, students were selected for inclusion in the study with a

target of 26 tenth graders within each school (Ingels et al., 2004). All tenth-grade students were

eligible for selection with the exception of foreign exchange students. Further stratifications were

made by race/ethnicity including Hispanic, Asian, Black, and Other. Asian students were

oversampled relative to White, African American, and Hispanic students in order to ensure a

minimum sample size for all subpopulations of students (Ingels et al., 2004). This sampling

approach yielded a nationally representative cohort of the tenth-grade student population in 2002
 37

(Ingels et al., 2014). Of 17,591 eligible students, 15,362 participated in the baseline wave of data

collection, yielding a weighted response rate of 87.3%.

In order to maintain representativeness, the ELS sample was refreshed during the first

follow-up in 2004 (Ingels et al., 2005). During this wave of data collection, an additional group

of twelfth-grade students was invited to participate in the study. Individuals in this group were

not in the tenth grade in the United States during the base year data collection for a variety of

reasons (e.g., enrolled in high school overseas, in an alternate sequence due to academic

performance, etc.). This refresh resulted in the addition of 238 participants to the overall sample

and a nationally representative cohort of twelfth-grade students (Ingels et al., 2014). Of 16,515

eligible students, 14,989 participated in the baseline wave of data collection, resulting in a

weighted response rate of 88.7%.

The base year sample along with the freshened cohort that was added during the first

follow-up formed the overall sample that was eligible for participation in the second follow-up in

2006 and third follow-up in 2012. No additional participants were added at either of these stages

although eligible respondents may have been temporarily out of scope, questionnaire ineligible,

or even rendered permanently ineligible for participation in subsequent questionnaire

administrations. Thus, of 15,892 respondents eligible for the second follow-up questionnaire,

14,159 participated, resulting in a weighted response rate of 88.4%. The sample for the third

follow-up questionnaire was further truncated. Of eligible 15,724 respondents, 13,250 completed

the third follow-up questionnaire, yielding a weighted response rate of 83.8%.

ELS Data Collection

 38

The ELS dataset contains variables collected from diverse sources over four rounds of

data collection across 15 years (Ingels et al., 2004). The majority of data was gathered directly

from the students via questionnaires, cognitive tests, and school records. Additional data was

collected from parents, teachers, and high school administrators via questionnaires. Thus, while

the student was the base unit of analysis given the longitudinal design of the study, it also was

designed to function as a multilevel study during the high school years. However, it is important

to acknowledge that the multilevel design does not extend to postsecondary enrollments.

Base year data collection began in 2002 which represented the sophomore year of high

school for the cohort. During this initial wave of data collection, self-administered baseline

questionnaires and cognitive tests in reading and mathematics were completed in group

administrations in the classrooms at their respective high schools (Ingels et al., 2004). Additional

questionnaires were completed by one parent of each student in the sample as well as the

student’s English teacher and mathematics teacher. For each participating high school, the

principal and director of the library media center completed a questionnaire during the base year

as well as a facilities checklist.

Subsequent data collection was directed primarily toward the student respondents

beginning with the first follow-up in spring 2004 (Ingels et al., 2005). The first follow-up

questionnaire was administered to students with distinct versions of the questionnaire for

students enrolled in the same high school as the base year, students who had transferred to a

different high school, students who had already completed high school, and students who had

dropped out of high school. While the majority of respondents completed self-administered

questionnaires in their high schools, select groups including early completers and dropouts were
 39

interviewed by telephone or in person. The cognitive test in mathematics was also administered

during the first follow-up. Finally, high school transcripts were requested from the participants to

document course taking from ninth grade through twelfth grade. During the first-follow-up, an

additional questionnaire was completed by school administrators.

The second follow-up wave of data collection occurred in spring 2006 (Ingels et al.,

2007). This sequencing was chosen with the intent that it would be two years after the scheduled

high school graduation of the initial cohort. It was the first wave of data collection in the ELS to

consist solely of a self-administered questionnaire. Multiple modalities were provided for

administration including web-based self-administration, interviewer administration of computer-

assisted telephone interview (CATI) interview, and computer-assisted personal interview

(CAPI).

Finally, the third and final follow-up wave of data collection for the ELS occurred in

summer 2012 (Ingels et al., 2014). Sequenced eight years after the scheduled high school

graduation of the initial cohort, this wave of data collection consisted of a student questionnaire

which, like the second follow-up, was administered via web-based self-administration, CATI, or

CAPI. Additionally, the third follow-up included collection of postsecondary transcripts during

spring 2015.

ELS Instrumentation

With its complex data structure, the ELS data source was comprised of numerous

instruments that supported data collection. Variables for the present study were derived from

data collected during the second, third, and fourth waves of data collection. A subset of variables

was extracted from the 2004 first follow-up questionnaire, the high school transcript, the 2006
 40

second follow-up questionnaire, the 2012 third follow-up questionnaire, and the postsecondary

transcript. The largest percentage of variables was selected from the third follow-up

questionnaire given the focus on postsecondary experiences and post-graduation college

outcomes.

Questionnaire Development

Items for each of the questionnaires were developed with protocols consistent with the

ELS research questions and priorities. Drawing on existing item pools from other NCES studies

including the National Assessment of Educational Progress (NAEP), NELS: 88, and the Program

for International Student Assessment (PISA), ELS instrumentation was designed based on the

following priorities (Ingels et al., 2004; Ingels et al., 2005; Ingels et al., 2007; Ingels et al.,

2014):

1. Items which could reasonably predict future outcomes of interest

2. Items which could achieve consistency with previous longitudinal studies

3. Items which reflect updates to theory and practice from prior longitudinal studies

4. Design choices that located items in questionnaire most likely to achieve accuracy

5. Elimination of items that required written parental consent consistent with the

Protection of Pupil Rights Act (PPRA) of 1978

Although the questionnaires at each stage of data collection differed in content, the first

three were developed and reviewed using the same general process (Ingels et al., 2004; Ingels et

al., 2005; Ingels et al., 2007). Questionnaire items were drafted by the study team and shared

with appropriate government agencies and other stakeholders. A Technical Review Panel (TRP)

comprised of content, methodological, and measurement experts was then empaneled to consider
 41

the questionnaires. Following the TRP review, staff members representing multiple divisions of

NCES reviewed questionnaires. Questionnaire items were then revised based on the feedback

gathered and justifications for the data elements composed. An additional round of review was

then conducted by the Office of Management and Budget (OMB). Following the revision

stemming from the OMB feedback, items were field tested and final revision enacted based on a

range of factors including item nonresponse, test-retest reliabilities, scale reliabilities, and

correlations between theoretically related measures.

The student questionnaire in the third and final follow-up wave of data collection in 2012

followed a slightly different process of development and revision (Ingels et al., 2014). Ideation

for this questionnaire began with a literature review that encompassed employment outcomes,

educational outcomes, and additional outcomes pertaining to adult development including living

arrangements, family composition, personal and professional values, and civic engagement. This

literature review informed the development of initial drafts which were reviewed across multiple

NCES departments. Recommendations for the inclusion of new topics as well as constructs were

then made by TRP representatives with expertise in psychology, economics, and sociology.

Newly developed items including social cognitive career theory (SCCT) items that were

developed specifically for the ELS were then tested via a series of cognitive interviews. Once the

revised drafts were reviewed by the TRP, justifications for the data elements were developed for

OMB review. Following this review, the final draft of the questionnaire was developed and field

tested before the questionnaire items were finalized and subject to final review by the OMB.

First Wave of Data Collection (2002)

 42

Completed in 2002, the first wave of ELS data collection consisted of a series of

questionnaires targeted to multiple audiences. With seven sections, the baseline student

questionnaire was designed to collect information about the tenth graders’ demographics, high

school experiences, future plans, language facility, work experience, and family dynamics

(Ingels et al., 2004). In addition to the student questionnaire, questionnaires were completed by

parents, teachers, and high school administrators. Despite the wealth of data available to

researchers from this wave of data collection, given the present study’s focus on postsecondary

experiences and outcomes, no variables from any of the questionnaires in the first wave of data

collection are included in the present study.

Second Wave of Data Collection (2004)

The second wave of ELS data collection in 2004 was more closely targeted to the student

audience, consisting of a student questionnaire as well as high school transcript collection. The

ELS first follow-up questionnaire consisted of eight sections, the first of which focused on

contact information to ensure continued access to respondents in future waves of data collection

(Ingels et al., 2005). Additional sections focused on the following areas: high school experiences,

time utilization, future plans, postsecondary planning, employment planning, current

employment, and support network including community, family, and friends. In addition to

perceptions of family support proximate to postsecondary enrollment and the degree to which

high school academic prepared students for college, the present study also drew on information

pertaining to respondent’s demographic characteristics and high school academic achievement,

primarily from the transcripts that were collected during the first follow-up wave of data

collection.
 43

Third Wave of Data Collection (2006)

The third wave of data collection was conducted in 2006. The student questionnaire in

this second follow-up was particularly targeted to considerations related to postsecondary access

and student decision making (Ingels et al., 2007). Although the questionnaire began with a

section on high school experiences, the majority of respondents in this wave of data collection

skipped it as they had previously responded to the first follow-up questionnaire and high school

transcript collection. Subsequent sections focused on early postsecondary experiences including

application, admission, financial aid, campus experiences, and educational expectations as well

as employment and community including family, living arrangements, and community

involvement. The section pertaining to early postsecondary experiences was particularly relevant

to the present study, providing important data points regarding first institution and experiences

while enrolled at that institution.

Fourth Wave of Data Collection (2012)

As with the second wave, the fourth wave of data collection consisted of a student

questionnaire coupled with a transcript collection, this time from postsecondary institutions at

which respondents had been enrolled as of 2012. The third follow-up student questionnaire

opened with a review of the respondent’s current status before transitioning to a more fulsome

examination of high school completion, postsecondary educational attainment, college

experiences, educational finances, educational expectations, employment and income, family

formation, and life values. Given the present study’s focus on the impact of participation in HIP

experiences on post-graduation college outcomes, this fourth wave of data collection provided a

substantial number of data points for inclusion in the model, most notably the respondent’s
 44

cumulative participation across multiple HIPs as well as all of the variables related to post-

graduation college outcomes including the perceived importance of postsecondary experiences in

preparation for various elements of adult life and perceived learning and challenge in

professional experiences.

Measures

All measures in the present study were derived from the public use ELS dataset with

variables from the first follow-up questionnaire administered in 2004, the second follow-up

questionnaire administered in 2006, and the third follow-up questionnaire administered in 2012.

Several variables were also used from the transcript data collections that supplemented the 2004

and 2012 questionnaires. In addition to the treatment and outcome measures, a total of 24

covariates were selected for inclusion in the model as shown in Appendix B.

The model for the present study included both observed and latent variables. Variables

that are measured by instruments such as surveys and tests are classified as observed variables

(Schumacker & Lomax, 2016). Observed variables can be derived directly from responses to

surveys and tests. As such, observed variables can be incorporated into SEM models individually

or as indicators of latent variables. In contrast to observed variables, latent variables cannot be

derived directly from responses to surveys or tests. Rather, they are measured indirectly using

responses from a set of questions or test items (Schumacker & Lomax, 2016; Wang & Wang,

2012). The use of both observed and latent variables in SEM models permits researchers to

account for substantive complexity in relationships between variables (Schumacker & Lomax,

2016; Streiner, 2006).

 45

Another way in which SEM affords researchers the ability to account for complex

relationships between observed and latent variables is via the inclusion of covariates into model

specification. Covariates are variables that are correlated with the independent variables in SEM

models, thereby influencing the relationship between the independent and dependent variables

(Schumacker & Lomax, 2016). Covariates can be observed or latent variables in SEM

(Schumacker & Lomax, 2016; Streiner, 2006). As shown in Appendix B, the model for the

present study included two latent covariates and 22 observed covariates.

Treatment Measure (HIPREC)

The treatment measure for the present study was a dichotomous measure of student

participation in two or more types of HIP experiences while enrolled as an undergraduate at a

postsecondary institution (HIPREC). This variable was calculated using ELS items F3A14A-

F3A14E9 with the item stem “Have you participated/Did you participate] in any of the following

as a part of your [undergraduate/college] education?” (National Center for Educational Statistics,

n.d.). HIP experience data collected during the 2012 third follow-up included (a) internship, co-

op, field experience, student teaching, or clinical assignment (F3A14A), (b) research project with

a faculty member outside of course or program requirements (F3A14B), (c) study abroad

(F3A14C), (d) community-based project as part of a regular course (F3A14D), and (e)

culminating senior experience, such as a capstone course, senior project or thesis, or

comprehensive exam (F3A14E) (National Center for Educational Statistics, n.d.). ELS variables

9
In this item set, students were also asked to indicate if they had ever participated in a program in which they have
had a mentor. To date, mentoring has not been specifically designated as a HIP in the existing scholarly discourse
(Kuh, 2008) so it is not included in this model as a treatment measure. That said, given that lack of a mentor has
been demonstrated to represent a barrier to HIP participation (Finley & McNair, 2013), it is included in the model as
a covariate. Given the lack of specification in the item stem around the nature of the mentor or mentoring program
(National Center for Educational Statistics, n.d., it was designated as a cocurricular experience (EXPCO-MEN).
 46

F3A14A-F3A15E were dichotomous indicators with response options as follows: no (0) and yes

(1). These variables were included in the third follow-up questionnaire administered in 2012.

For the treatment measure (HIPREC) in the present study, sample members were coded

as 0 if they responded yes to fewer than two of the ELS F3A14A-F3A15E items. They were

coded as a 1 if they responded yes to two or more of the ELS F3A14A-F3A15E questions.

Within the analytic sample, the group of respondents who reported two or more HIP experiences

(1) during their undergraduate enrollment were designated as the treatment group as they

completed or exceeded the recommended number of HIP experiences (Kuh, 2008).

Correspondingly, the group of respondents who indicated that they had less than two HIP

experiences (0) during their undergraduate career were designated as the control group.

Outcome Measure

The latent outcome measure for Manuscript I represented respondent’s perception of the

importance of their postsecondary experiences for preparation for adult life (PREP). This

variable is derived from ELS items: F3A15A-F3A15D with the item stem “How important

would you say your [undergraduate education was in/college education was in/college

attendance has been in] preparing you for the following aspects of your life?” (National Center

for Educational Statistics, n.d.). Collected via the 2012 third follow-up questionnaire, F3A15A

targeted work and career, F3A15B denoted further education, F3A15C financial security, and

F3A15D indicated civic participation (National Center for Educational Statistics, n.d.). These

items were measured on a three-point scale: very important (1), somewhat important (2), and not

at all important (3). For purposes of this analysis, the items were reverse coded to reflect

increasing levels of importance.

 47

The latent outcome measure for Manuscript II represented perceived learning and

challenge in professional experiences (LEARN). This variable was derived from ELS items:

F3B25B, F3B25D, and F3B25F with the item stem “Please indicate to what extent the following

job characteristics [apply/applied] to your [current/most recent job].” (National Center for

Educational Statistics, n.d.). Collected via the 2012 third follow-up questionnaire, F3B25B

corresponded with opportunities to learn new things, F3B25D indicated new challenges, and

F3B25F denoted societal impact (National Center for Educational Statistics, n.d.). These items

were measured on a five-point scale that ranged from definitely not an aspect of the respondent’s

current/most job (1) to very much an aspect of the respondent’s current job (5).

Observed Covariates

The observed covariates in the present study represented a variety of factors including

student precollege characteristics, organizational factors, and individual student experiences that

have been identified as influential to educational attainment in contemporary theoretical

frameworks related to college outcomes (Reason, 2009; Terenzini & Reason, 2005). Within the

category of precollege characteristics, three of these indicators related to sociodemographic

characteristics, one represented academic preparation and performance in high school, and one

pertained to student dispositions. Organizational indicators included two covariates that

represented institutional structural-demographic characteristics while individual student

experience indicators included seven covariates related to students’ curricular experiences and

eight related to students’ co-curricular experiences. Please see Appendix B for a list of all

covariates included in the model.

Student Respondent Race/Ethnicity Covariate (SCSD-RACE)

 48

This covariate was informed by the ELS F1RACE variable, a composite variable derived

from student responses in the 2004 first follow-up questionnaire (National Center for

Educational Statistics, n.d.). It was a self-report measure of the respondent’s race/ethnicity

derived from student responses in the 2002 base year questionnaire (BYRACE) or imputed if

missing (National Center for Educational Statistics, n.d.). The measure consisted of a summation

of responses to a series of items with dichotomous scales (F1RACE) related to race/ethnicity

(National Center for Educational Statistics, n.d.). Response scales for the original F1RACE

variable were as follows: American Indian/Alaska Native, non-Hispanic (1), Asian,

Hawaii/Pacific Islander, non-Hispanic (2), Black or African American, non-Hispanic (3),

Hispanic, no race specified (4), Hispanic, race specified (5), More than one race, non-Hispanic

(6), and White, non-Hispanic (7).

These ELS F1RACE response scales were used to create a dichotomous covariate

(SCSD-RACE), which specified the racial/ethnic identity of the student respondent as a member

of a minoritized population within postsecondary educational institutions in the U.S. (1) or a

member of an overrepresented population within postsecondary educational institutions in the

U.S. (0). For purposes of this analysis, the original ELS categories of American Indian/Alaska

Native, non-Hispanic; Black or African American, non-Hispanic; Hispanic, no race specified;

Hispanic, race specified; and More than one race, non-Hispanic were recoded as members of

minoritized populations (1)10. The original ELS categories of Asian, Hawaii/Pacific Islander,

10
It is important to acknowledge that the aggregation of these distinct racial/ethnic populations results in limitations
to the conclusions that can be drawn in the present study (McNair et al., 2020), particularly in the context of the
student experiences and college outcomes of minoritized student populations in the U.S. (Patton et al., 2015).
However, these limitations are discussed in Chapters 4 and 5 along with the relative lack of racial/ethnic diversity in
the analytic sample.
 49

non-Hispanic and White, non-Hispanic were coded as members of non-minoritized populations

(0).

Student Respondent’s Biological Sex Covariate (SCSD-SEX)

This observed covariate was informed by the ELS F1SEX variable, a composite variable

derived from student responses in the 2002 base year questionnaire (BY1SEX) (National Center

for Educational Statistics, n.d.). It was a self-report measure of the respondent’s biological sex.

Response scales for the original F1SEX variable were as follows: Male (1) and Female (2). For

purposes of this analysis, the original ELS category of Female was recoded as 0.

Student Respondent’s Socioeconomic Status (SES) Covariate (SCSD-SES)

This observed covariate was comprised of the ELS F1SES2 variable, a representation of

the student respondent’s family SES as of their senior year of high school. This was a composite

continuous measure which included indicators of total family income from all sources in the base

year and the highest level of education attained by the student’s mother and father along with

measures of occupational prestige for both the mother and father as of the 2004 first-follow-up

(National Center for Educational Statistics, n.d.).

Student High School Grade Point Average (GPA) Covariate (SCHS-GPA)

This observed covariate was informed by the ELS F1RGPP2 variable, a categorical

variable derived from the continuous cumulative GPA variable for all courses taken from ninth

grade through twelfth grade (National Center for Educational Statistics, n.d.). With recent

scholarship offering evidence of a positive and consistent relationship between high school GPA

and college graduation (Adelman, 2006; Allensworth and Clark, 2020), this measure served as an

indicator of student’s prior academic achievement. It was part of the high school transcript data
 50

collection during the first follow-up in 2004. Response scales for the ELS F1RGPP2 variable

were as follows: 0.00 - 1.00 (0), 1.01 - 1.50 (1), 1.51 - 2.00 (2), 2.01 - 2.50 (3), 2.51 - 3.00 (4),

3.01 - 3.50 (5), and 3.51 - 4.00 (6). Although the original ELS variable was categorical, the

presence of six categories ordered to reflect increasing levels of academic achievement permitted

this measure to be treated as a continuous variable for purposes of the present study.

Student Educational Goals Covariates (SCD-GOAL)

This observed covariate was informed by the ELS F1STEXP variable, a composite

variable derived from student responses in the 2004 first follow-up questionnaire (National

Center for Educational Statistics, n.d.). It was a self-report measure of the highest level of

education that the student respondent expected to achieve as of 2004. Given prior scholarship

documenting the positive relationship between academic goals and college persistence (Brown et

al., 2008; Robbins et al., 2004), this measure served as a dispositional indicator in the model

(Reason, 2009). Item response scales were as follows: less than high school graduation (1), GED

or other equivalency only (2), high school graduation only (3), attend or complete 2-year

college/school (4), attend college, 4-year degree incomplete (5), graduate from college (6),

obtain Master's degree or equivalent (7), obtain PhD, MD, or other advanced degree (8), and

don't know (9). For purposes of this analysis, the don’t know (9) response option was recoded as

a 0, reflecting an absence of ability to define an educational goal at the time of measure

(Lombardi et al., 2011). Although the original ELS variable was categorical, the presence of nine

categories ordered to reflect increasing levels of educational goals allowed this measure to be

treated as a continuous variable in the present study.

Control of First Postsecondary Institution Covariate (ORG1-CON)

 51

This observed covariate was informed by the ELS F3PS1SEC variable, a categorical

variable which represented the control and level of the first postsecondary institution attended as

defined by the Integrated Postsecondary Education Data System (IPEDS) (National Center for

Educational Statistics, n.d.). The F3PS1SEC was collected during the third follow-up in 2012

and was included in the model consistent with scholarship that signals that first institution may

influence educational attainment (Pascarella & Terenzini, 2005). Response options for this item

were as follows: four-year public (1), four-year private, not for profit (2), four-year private for-

profit (3), two-year public (4), two-year private for-profit (6), and other (7). For purposes of this

analysis, these categories were collapsed into a dichotomous indicator of institutional control

with four-year public and two-year public recoded as public institutions (0). Four-year private,

not for profit; four-year private for-profit; and two-year private for-profit were recoded as private

institutions (1). Responses in the other category were coded as missing (-9).

Level of First Postsecondary Institution Covariate (ORG1-LEV)

This observed covariate was informed by the ELS F3PS1SEC variable, a categorical

variable which represented the control and level of the first postsecondary institution attended as

defined by the Integrated Postsecondary Education Data System (IPEDS) (National Center for

Educational Statistics, n.d.). Selected for inclusion given scholarship that documents that choice

of first institution influences educational attainment (Pascarella & Terenzini, 2005), the

F3PS1SEC data was collected during the third follow-up in 2012. Response options for this item

were as follows: four-year public (1), four-year private, not for profit (2), four-year private for-

profit (3), two-year public (4), two-year private for-profit (6), and other (7). For purposes of this

analysis, these categories were collapsed into a dichotomous indicator of institutional level with
 52

two-year public and two-year private, for-profit recoded as two-year institutions (0). Four-year

public; four-year private, not for profit; and four-year private for-profit were recoded as four-

year institutions (1). Responses in the other category were coded as missing (-9).

Transfer Status Relative to First Institution Covariate (EXPCU-TSFR)

This observed covariate was derived from the F3PS1RETAIN variable, a categorical

variable indicating the student respondent’s relationship to their first postsecondary institution

(PS1) at the time of the third follow-up in 2012 (National Center for Educational Statistics, n.d.).

Response options for the F3PS1RETAIN item were as follows: earned a credential from PS1,

still attending PS1 as of F3 (1); earned a credential from PS1, no longer attending PS1 as of F3

(2); no credential from PS1, still attending PS1 as of F3 (3); no credential from PS1, no longer

attending PS1, did attend another PS institution (4); and no credential from PS1, no longer

attending PS1, did not attend another PS institution (5). For purposes of this analysis, these

categories were recoded into a dichotomous indicator of transfer status. Respondents who

selected options 4-5 on the original F3PS1RETAIN scale were recoded as transfers (1) while

respondents who selected options 1-3 were recoded as non-transfers (0).

Curricular Field of Study Covariates

This set of observed covariates was informed by the ELS F3TZBCH1CIP2 variable, a

categorical variable based on the 2-digit 2010 NCES Classification of Instructional Programs

(CIP) code for the first known bachelor's degree (National Center for Educational Statistics,

n.d.). This variable was part of the postsecondary transcript data collection during the third

follow-up in 2012. Consistent with recent scholarship that identified interaction effects of HIP

participation on post-graduation college outcomes that varied by student major (Wolniak &
 53

Engberg, 2019), the original F3TZBCH1CIP2 variable was recoded to aggregate the fields of

study into six categories with specifications as follows: Arts and Humanities (EXPCU-ART),

Business (EXPCU-BUS), Education (EXPCU-EDU), Health and Human Services (EXPCU-

HHS), Social Science (EXPCU-SS), and Science, Technology, Engineering, and Math (STEM)

(EXPCU-STEM). Please see Appendix C for a comprehensive list of the F3TZBCH1CIP2 CIP

codes that are contained within each category. As shown in Appendix C, these categories were

used to create a set of six dichotomous covariates which served as indicators of the academic

field of study for the student’s first known bachelor’s degree.

Cocurricular Experiences at First Postsecondary Institution Covariates

This set of observed covariates was informed by the ELS F2B18 variable with the item

stem “During the time that you have been enrolled at [first attended postsecondary institution

(F2PS1)], how often have you participated in the following activities?” (National Center for

Educational Statistics, n.d.). Activities noted in the item stems included talking with faculty

outside of class (F2B18A), meeting with their advisor about academic plans (F2B18B),

participating in intramural or nonvarsity sports (F2B18E), participating in varsity or

intercollegiate sports (F2B18F), and participating in extracurricular activities (F2B18G). The

ELS F2B18 item was part of the student questionnaire during the second follow-up in 2006.

Response options for these indicators were as follows: never (1), sometimes (2), and often (3).

As shown in Appendix B, these dichotomous covariates (EXPCO-FAC, EXPCO-ADV, EXPCO-

INT, EXPCO-VAR, EXPCO-EXT) served as indicators of the cocurricular experiences in which

the student respondent had participated as of the second-follow-up data collection.

Student Respondent’s Employment Status Covariate (EXPCO-JOB1 and EXPCO-JOB2)

 54

These observed covariates were derived from two variables included in the second

follow-up questionnaire in 2006: F2C24_P which indicated the number of jobs held during the

2004-2005 school year (EXPCO-JOB1) and F2C29_P which denoted the number of jobs held

during the 2005-2006 school year (EXPCO-JOB2). Both of these variables were categorical and

measured on the following scale: no jobs (0), one job (1), two jobs (2), three jobs (3), and four or

more jobs (4). The original measurement scales were retained for the present study.

Student Respondent’s Access to Mentoring Experience (EXPCO-MEN)

This observed covariate was informed by the ELS F3A14F variable, a dichotomous

indicator derived from student responses in the third follow-up questionnaire in 2012. It was a

self-report measure of student participation in a program in which they had a mentor as part of

their undergraduate/college education (National Center for Educational Statistics, n.d.).

Response options for the original variable consisted of no (0) and yes (1) and were retained for

the present study.

Latent Covariates

The latent covariates in the present study represented two perceptual measures that relate

primarily to student precollege characteristics (Reason, 2009; Terenzini & Reason, 2005). The

first of these variables placed particular emphasis on support from family and friends for college

enrollment as of the first follow-up in 2004, an area that Reason (2009) particularly noted as

warranting further exploration. The second latent covariate was a measure of perceived academic

preparedness for enrollment at the student’s first postsecondary institution at the time of the

second follow-up in 2006.

Student Respondent’s Perception of Support Network Aspirations (SCSD-FAMSUP)

 55

The perception of support latent variable targeted respondent perceptions of their support

network and the degree to which members of that network support postsecondary enrollment.

This variable was derived from ELS items: F1S44A-F1S44D with the item stem “What do the

following people think is the most important thing for you to do right after high school?”

(National Center for Educational Statistics, n.d.). Collected via the 2004 first follow-up

questionnaire, F1S43A corresponded with the student respondent’s perception of their mother’s

aspirations while F1S43B indicated the student respondent’s perception of their father’s

aspirations (National Center for Educational Statistics, n.d.). Similarly, F1S43C corresponded

with the student respondent’s perception of an unspecified close relative’s aspirations while

F1S43D indicated the student respondent’s perception of an unspecified friend’s aspirations

(National Center for Educational Statistics, n.d.). Each of these variables shared same response

scale: go to college (1), get a full time job (2), enter vocation or apprenticeship (3), enter military

service (4), get married (5), they think I should do what I want (6), and don't know (7). For

purposes of this analysis, these variables were recoded to reflect increasing support for

postsecondary enrollment. The don’t know response option was recoded as 0 to reflect an

absence of knowledge about the aspirations of the respondent’s family and friends (Lombardi et

al., 2011). The response option noting that respondents should do what they want to do was

recoded as a 1. Support for all other non-college options including a full-time job, vocation or

apprenticeship, military service, and marriage were collapsed into a single non-college category

(2). The college enrollment response option was recoded as college enrollment (3).

Student Respondent’s Perception of Academic Preparedness (SCHS-PREP)

 56

The perception of academic preparedness latent variable indicated respondent perceptions

of the degree to which their high school class experiences prepared them for enrollment at their

first postsecondary institution. This variable is derived from ELS items F2B17A-F2B1C with the

item stem “To what extent did the following high school courses prepare you for [first attended

postsecondary institution (F2PS1)]??” (National Center for Educational Statistics, n.d.).

Collected via the 2006 second follow-up questionnaire, F2B17A corresponded with high school

math courses, F2B17B with high school science courses, and F2B17C with high school

English/writing courses (National Center for Educational Statistics, n.d.). All indicators shared

the same response scale: not at all (1), somewhat (2), and a great deal (3), all of which were

retained for purposes of this analysis.

Weights

In order to account for the complexities of the longitudinal design including unequal

patterns of selection and nonparticipation from selected sample members (Ingels et al., 2014), the

sampling weight F3FITSCWT was applied consistent with NCES guidelines noted in the ELS

technical documentation (Ingels et al., 2014). These guidelines specify that the F3FITSCWT

weight be used in analyses with the G12COHORT as the population under study and in analyses

that include variables from the first follow-up questionnaire, the high school transcript data

collection, the second follow-up questionnaire and the third follow-up questionnaire (Ingels et

al., 2014).

Additionally, in the present analysis, balanced repeated replication (BRR) weight

adjustments were needed to facilitate accurate estimations of standard errors (McNeish et al.,

2017; Stapleton, 2008). In order to support these adjustments, six sets of 200 BRR weights were
 57

computed by NCES consistent with the agency’s guidelines (Ingels et al., 2014). This guidance

further specified that F3F1T001-F3F1T200 BRR set corresponded with the selected

F3FITSCWT sampling weight.

Data Analysis

The present study employed structural equation modeling (SEM) to conduct propensity

score (PS) analysis using MPlus software (Muthén & Muthén, 2017). With the aim of reducing

selection bias associated with the opt-in nature of the HIPREC treatment measure, the study

design followed an eight-step analytic approach proposed and tested by Leite et al. (2019) and

described in detail in this section.

Data Preparation

In this initial stage of model development, the ELS public use data file was examined to

identify observed and latent covariates that relate to both the HIPREC treatment measure and the

college outcomes measures. Hypothesized to be true confounders in the model (Steiner et al.,

2010), the inclusion of these covariates was consistent with contemporary college outcomes

theory (Reason, 2009; Terenzini & Reason, 2005). As shown in Appendix B, a total of 24

covariates were identified: five observed covariates pertaining to precollege characteristics (three

sociodemographic, one academic preparation, one dispositional); two latent covariates pertaining

to precollege characteristics (one family support, one academic preparation); two observed

covariates pertaining to postsecondary institution organizational context, all of which were

structural demographic in nature; and 15 observed covariates representing individual student

experiences (seven curricular and eight cocurricular).

 58

With the exception of the sociodemographic student characteristics such as race/ethnicity,

sex, and SES, many of the model variables had some missing data. The nature of the missingness

was attributed to a variety of factors including item nonresponse, survey skip logic, item skip

logic, and even non-administration (National Center for Educational Statistics, n.d.). For this

analysis, the first step in the data management procedure was to review data that was coded as

missing for any reason and recode as missing (-9) to facilitate calculation of the percentage of

missing data and evaluate missing data patterns. Within the dataset for Manuscript I, 37.29% of

the cases had data missing on at least one variable with 148 missing data patterns present. For

Manuscript II, 38.66% of cases had data missing on at least variable with 155 missing data

patterns.

The issue of missing data in survey datasets related to postsecondary experiences and

outcomes is not uncommon (Cheema, 2014). The use of techniques to address missing data such

as expectation-maximization (EM), full-information maximum likelihood (FIML) and multiple

imputation (MI) for such datasets is typically encouraged (Cox et al., 2014). A particular

advantage of both EM and MI is the preservation of the sample size of the covariates (Leite et

al., 2019).

That said, there are limitations to the use of MI in the current study given both the nature

of the data structure and the complexity of the analytic techniques. While multiple studies have

addressed the issue of combining PS estimates generated by MI (Hill, 2004; Mitra & Reiter,

2016), there is little empirical support for procedures to combine fit indices generated by MI

(Enders & Mansolf, 2016). Additionally, the categorical nature of multiple covariates in the

proposed models precluded the use of EM and FIML at distinct points in the analytic process.
 59

Thus, while techniques such as EM, FIML, and MI were attempted in order to address issues of

missing data, full case analysis (i.e., listwise deletion) was ultimately adopted. While not the

most sophisticated approach currently available to researchers, full case analysis can be

reasonably employed under the right conditions (Cox et al., 2014) related to the nature of the

missingness (Allison, 2002) and sufficiency of included covariates (Graham, 2009).

Step 1: Multiple-Group Confirmatory Factor Analysis (CFA) for Latent Covariates

Following the data preparation, a multiple-group CFA was fit in MPlus (Muthén &

Muthén, 2017) to verify the factor structure of the latent covariates (Schumacker & Lomax,

2016; Wang & Wang, 2012). It was run with the two latent covariates (SC-FAMSUP and SCHS-

PREP) and the treatment and control groups. Due to the categorical nature of many of the

covariates in the model, this multiple-group CFA was fit with weighted least squares with

adjusted means and variances (WLSMV) estimation (Beauducel & Herzberg, 2006). Models

were run with and without replication weights. While the models were evaluated for exact fit

using a modified Chi-Square (χ2) (Stapleton, 2008), conventional practice in CFA stipulates that

multiple indices be used to assess fit (Schumacker & Lomax, 2016). Fit indices recommended by

Leite et al. (2019) for these purposes included the Comparative Fit Index (CFI), the Tucker-

Lewis Index (TLI), and the Root-Mean-Square Error of Approximation (RMSEA). For this

analysis, fit was determined to be close if CFI  0.95, TLI  0.95, and RMSEA  0.05.

An additional purpose of this multiple-group CFA was to evaluate measurement

equivalence of the latent covariates across treatment and control groups. Known as scalar

invariance (Millsap & Olivera-Aguilar, 2012), it is a required condition for mitigating selection

bias in PS analysis (Leite et al., 2019), thereby permitting appropriate comparison of means and
 60

variances across the treatment and control groups. Scalar invariance was tested with an adjusted

likelihood ratio test in MPlus (Muthén & Muthén, 2017). In this case, Chi-Square (χ2) served as

the relevant test statistic; an absence of statistical significance indicated that the assumption of

scalar invariance had been met.

Step 2: SEM for PS Estimation

Once the factor structure and scalar invariance was confirmed for the latent variables, a

SEM was fit in MPlus (Muthén & Muthén, 2017) to calculate PS weights. Please see Appendix

D for the conceptual model for this step in the analytic process. The SEM was run with the latent

and observed covariates as predictors and the HIPREC treatment measure as the outcome. Due to

the dichotomous nature of many of the covariates in the model, the SEM was fit with WLSMV

estimation (Muthén & Muthén, 2017). As with the previous step, the model was evaluated for

exact fit using a modified Chi-Square (χ2) (Stapleton, 2008) as well as CFI, TLI, and RMSEA fit

indices. For this analysis, fit was determined to be close if CFI  0.95, TLI  0.95, and RMSEA

 0.05.

Given that the literature suggests a cumulative benefit of HIP participation to participants

in increasing numbers of HIP experiences (Finley & McNair, 2013; McDaniel & Van Jura, 2020;

Myers et al., 2018), PS weights were estimated using average treatment effect on the treated

(ATT) weights (Heckman, 2005). The PS weights for the treatment and control groups were then

compared to evaluate the assumption of overlap in PS analysis which is designed to assess

whether the sample is derived from one population or two populations. For this analysis, the

assumption of overlap was determined to be have been met if neither group had a PS with a
 61

value of 0 which would indicate no probability of treatment or 1 which would signal 100%

probability of treatment (Leite et al., 2019).

Step 3: Latent Covariate Balance Evaluation

Following the calculation of the PS weights, these values were multiplied by the

F3FITSCWT sample weight to obtain final PS weights. Covariate balance was evaluated using

regression models of the latent covariates on the treatment measure. This assessment permitted

the researcher to compare the distributions of the treatment and control groups following

application of the final PS weights via examination of the standardized mean differences across

the regression coefficients (Leite et al., 2019). This assessment was performed using two sets of

criteria given a lack of consensus in the literature. Austin (2011) stipulates absolute values of

standard mean differences below 0.100 standard deviations as evidence of adequate covariate

balance while Stuart and Rubin (2007) designate .250 as an appropriate threshold by which to

evaluate covariate balance.

Step 4: Observed Covariate Balance Evaluation

In addition to evaluating the balance of the latent covariates, covariate balance of the

observed covariates was also evaluated using regression models of the observed covariates on

the treatment measure. As with the latent covariate balance evaluation, this procedure compared

the distributions of the treatment and control groups following application of the final PS weights

(Leite et al., 2019). The regression models were weighted with the calculated PS weights for the

ATT. Once again, the standardized regression coefficients were used to evaluate the adequacy of

the covariate balance using the standardized mean difference calculations (Leite et al., 2019). As
 62

with Step 3, covariate balance was evaluated using criteria established by Austin (2011) and

Stuart and Rubin (2007).

Step 5: Multiple-Group CFA for Latent Outcomes Using PS Weights

Following the estimation of the PS weights, a multiple-group CFA was fit in MPlus

(Muthén & Muthén, 2017) for the outcome measures using the final PS weights with the sample

weight adjustments. These analyses were performed with the adjusted latent outcome variables

and the treatment and control groups using WLSMV estimation. This CFA was used to affirm

the factor structure of the latent outcome variables (Schumacker & Lomax, 2016; Wang &

Wang, 2012). Scalar invariance (Millsap & Olivera-Aguilar, 2012) of the outcome measure

across the treatment and control groups was also evaluated using the adjusted likelihood ratio test

(Muthén & Muthén, 2017). As with the previous scalar invariance analysis, Chi-Square (χ2)

served as the relevant test statistic with an absence of statistical significance indicating that the

assumption of scalar invariance has been met.

Step 6: Multiple-Group CFA for ATT Estimation Using PS Weights

In addition to testing for scalar invariance, the multiple-group CFA was used to estimate

the ATT using PS weights. As noted in Step 5, this model was run with the adjusted latent

outcome variable and the treatment and control groups using WLSMV estimation. The mean of

the outcome measure for respondents with no HIP experiences was set to zero to identify the

model. With this adjustment, the resultant mean of the outcome measures for respondents who

have participated in two or more HIP experience represented the latent mean difference between

the groups, i.e., the ATT (Leite et al., 2019). Models were run with and without replication

weights. Once again, the model was evaluated for exact fit using a modified Chi-Square (χ2)
 63

(Stapleton, 2008) as well as with CFI, TLI, and RMSEA fit indices. For this analysis, fit was

determined to be close if CFI  0.95, TLI  0.95, and RMSEA  0.05.

Step 7: Multiple-Group SEM for Interaction Effect Estimation

A multiple-group SEM was used to evaluate interactions between the outcome measure

and the following observed covariates: race/ethnicity (SCSD-RACE), sex (SCSD-SEX), SES

(SCSD-SES), and high school GPA (SCHS-GPA). This analytic step permitted the researcher to

consider whether the ATT remained the same across the subpopulations (Wang & Wang, 2012).

The estimated differences for the coefficients between the groups were used to determine the

nature of any identified interactions. The conceptual model for Manuscript I is depicted in

Appendix E while the conceptual model for Manuscript II is included as Appendix F. To

evaluate the interactions, covariate coefficients were set to vary across groups (Leite et al.,

2019). Although no covariates were needed due to the presence of the final PS weights, linear,

cubic, and quadratic PS estimates were added to achieve doubly robust estimation and remove

any remaining bias (Bang & Robins, 2005; Schafer & Kang, 2008).

Step 8: Multiple-Group SEM with Phantom Variable for Sensitivity Analysis

Using the PS weights, a multiple-group SEM with a phantom variable was fit in MPlus

(Muthén & Muthén, 2017) to evaluate in the sensitivity of the model to omitted confounders

consistent with the method specified by Harring et al. (2017). Aligned with Leite et al. (2019),

nine multiple-group SEMs were used with varying permutations of sensitivity parameters. The

first parameter which corresponded with SMD on the phantom variable was tested with

parameters of 0.25, 0.5, and 1.0. It was tested in combination with the second parameter, which

consisted of the standardized path from the phantom variable to the outcome measure, likewise
 64

with parameters of 0.25, 0.5, and 1.0. If the p values for any of the interactions across the

multiple-group SEMs demonstrated a change in statistical significance (i.e., from non-

significance to statistical significance) following the parameter adjustment, it was interpreted to

signal the absence of potential confounders.

Conceptual Models

The specification of the models that were tested using the aforementioned methodology is

represented via path diagram conventions for representing the variables within the models

(Schumacker & Lomax, 2016). The conceptual model for PS estimation is included as Appendix

D while the respective conceptual models for the multiple-group SEM for Manuscript I and

Manuscript II are presented in Appendices E and F respectively.

Conceptual Model for PS Estimation

Appendix D depicts the conceptual model that corresponds with PS Estimation, the

second step in the methodology outlined above (Leite et al., 2019). This model is hypothesized to

represent variables that have been demonstrated to be correlated with participation in HIP

experiences in the college outcomes literature. The inclusion of these covariates was designed to

mitigate the issue of selection bias associated with the opt-in nature on HIP experiences (Steiner

et al., 2010). In the diagram shown in Appendix D, the two latent covariates (SCSD-FAMSUP

and SCHS-PREP) are represented with ovals in the path diagram while each of the 22 observed

covariates is represented as a rectangle consistent with path diagram conventions (Schumacker &

Lomax, 2016).

Conceptual Models for Multi-Group SEM

 65

The conceptual models in Appendices E and F present the hypothesized models that were

tested in the Multiple-Group SEM for Interaction Estimation, the seventh step in the analysis

(Leite et al., 2019). In these models, participation in two or more HIP experiences served as a

grouping variable with sample members who completed two or more HIPs considered to be the

treatment group and those who completed fewer than two HIPs designated as the control group.

Depicted as rectangles in Appendices E and F, linear, cubic, and quadratic PS estimates were

included to achieve doubly robust estimation (Bang & Robins, 2005; Schafer & Kang, 2008).

Consistent with path diagram conventions (Schumacker & Lomax, 2016), the ovals signal that

the PREP outcome for Manuscript I and the LEARN outcome for Manuscript II were both latent

variables. In both manuscripts, interaction effects were tested for student membership in a

minoritized racial population (SCSD-RACE), student sex (SCSD-SEX), student SES (SCSD-

SES), and high school GPA (SCSD-GPA). As observed covariates, these variables were depicted

as rectangles consistent with path diagram conventions (Schumacker & Lomax, 2016).
 66

CHAPTER IV

MANUSCRIPT I

Abstract

This study examines the relationship between cumulative participation in high-impact

practices (HIP) and perceived importance of postsecondary experience in preparation for adult

life using the Educational Longitudinal Study of 2002 (ELS) dataset. Using a methodology

proposed and tested by Leite et al., (2019), this analysis incorporated the ability to account for

self-selection into HIPs using propensity score (PS) analysis with a multiple-group structural

equation model (SEM) design to examine differences between students who participated in two

or more HIPs and those who did not (n = 3,105). Results offered evidence of benefit to

participation in two or more HIP experiences with positive and statistically significant

differences in perceived importance of postsecondary education in preparation for adult life

across the analytic sample with doubly robust estimation techniques. Interaction effects for

female students, students from low SES backgrounds, and students who are members of

minoritized racial/ethnic populations were also identified. The findings offered evidence of post-

graduation impact to participation in HIPs that can support program development and student

decision making as well as support the future use of analytic techniques such as PS analysis,

doubly robust estimation, and sensitivity analysis.

Introduction

Since 2007, the scholarly discourse on college outcomes has foregrounded a set of

undergraduate student experiences that have been termed high-impact practices (HIPs) (Kuh,

2008). Elevated by the American Association of Colleges and Universities (AAC&U) via the
 67

Liberal Education and America’s Promise initiative (2007), this set of undergraduate educational

experiences are characterized sustained independent inquiry, student interaction with faculty and

peers, and opportunities for regular feedback, synthesis, and perspective taking (Kuh, 2008). HIP

experiences include first-year seminars and experiences, common intellectual experiences (i.e.,

core curriculum), learning communities, writing-intensive courses, collaborative assignments

and projects, undergraduate research, diversity and global learning, service learning and

community-based learning, internships, and capstone courses and projects (American

Association of Colleges and Universities, 2007; Kuh, 2008).

Even as foundational scholarship acknowledged that there are limits to what

postsecondary institutions can do to improve college outcomes for all students, particularly those

who come to these institutions with years of educational disadvantage (Kuh, 2008), the earliest

HIP scholarship (Kinzie et al., 2008; Kuh, Kinzie, Buckley, et al., 2007; Kuh, Kinzie, Cruce, et

al., 2007; Kuh, 2008; Kuh et al., 2008) identified small but significant impacts from participation

in HIP experiences on college outcomes including academic achievement and persistence from

the first year of enrollment to the second year

Building on this foundational work, the nature of the relationship between high-impact

practices (HIPs) and college outcomes has continued to be explored and refined using diverse

institutional and multi-institutional datasets. However, since 2008, postsecondary institutions

have experienced considerable disruptions, resulting in shifts in both enrollment and financial

support structures. The first of these impacts was experienced following the Great Recession

(Barr & Turner, 2013) which closely followed the earliest HIP inquiry and its emergence in the

college outcomes scholarship. More proximately, the impacts of the COVID-19 pandemic are
 68

now just beginning to be experienced by postsecondary institutions (Friga, 2021) and much

remains to be understood about longer term implications. Moreover, recent surveys indicate that

confidence in the value of higher education may be eroding due to the pandemic with current

college students and future college students reporting increasing doubts about whether

postsecondary education is worth the cost (Klebs et al., 2021).

Thus, it is critical that nuances and limitations in the conclusions that can be drawn from

the literature regarding HIP participation be fully explored to ascertain more precisely who

benefits from HIP experiences, to what extent, and under what conditions are those impacts more

likely to realized. This critical examination is intended to support the ability of postsecondary

institutions to deploy limited financial resources efficiently and effectively in a manner that

maximizes the transformative potential of student engagement. A chronological examination of

the HIP scholarship offers valuable insight into how datasets, models, and methods have evolved

over time, resulting in increased complexity in terms of research design, data analysis, and

actionable insights afforded by the results while also illuminating gaps in the current scholarship

that warrant further study.

Theoretical Framework

The HIP discourse has drawn heavily on traditional models of college outcomes and

student success (Astin, 1984; Astin, 1993; Tinto, 1993; Terenzini et al., 1996). With an emphasis

on the choices that students make regarding how to direct their time and energy during their

respective undergraduate enrollments (Astin, 1984), the interplay between precollege

characteristics, postsecondary experience, and postgraduation outcomes has long been

acknowledged. Astin’s Inputs-Environment-Outputs (I-E-O) model (1993) conceptualized the

 69

inputs as student characteristics at time of entry; the environment as the people, programs, and

experiences with which the student engages as an undergraduate; and the outputs as the student

characteristics at time of departure. Tinto’s Student Departure Model (1993) offered a similar

conceptualization of this synergy. In this model, pre-entry attributes, intended and actualized

student goals and commitments, and academic and social experiences while enrolled served as a

framework for student retention. Terenzini et al. (1996) further distilled these constructs, noting

that six primary elements to be central to understanding college outcomes: precollegiate traits,

curricular patterns, in-class experiences, out-of-class experiences, institutional context, and

learning outcomes.

More recently, Reason (2009) offered a more expansive view of college outcomes with a

particular focus on student persistence. Drawing on a collaboration between Terenzini and

Reason (2005), this framework likewise foregrounded the importance of precollege

characteristics and experiences that include sociodemographic characteristics as well as family

support albeit with the acknowledgement that family support is underexplored in current

scholarship. Like Tinto (1993), Reason (2009) separated student dispositions such as goals and

motivation and academic preparation in high school from precollegiate sociodemographic

characteristics. The importance of the organizational context of the postsecondary institution is

reflected by the inclusion of structural-demographic characteristics such as institutional control,

size, and selectivity with a particular emphasis on first institution attended consistent with the

findings of Pascarella and Terenzini (2005). However, Reason (2009) cautioned against limiting

an examination of institutional context to structural-demographic characteristics, noting that

organizational behavior dimensions including institutional priorities, campus policies, and

 70

decision-making structures are equally important, a consideration affirmed by subsequent

scholarship (Ro et al., 2013). Finally, Reason (2009) offered important distinctions across

student experiences, separating the peer environment including the racial and academic climate

from individual student experiences which can occur in the classroom; in the curriculum via

student major and participation in HIPs; and outside the classroom via participation in programs

and activities such as athletics and student organizations.

Literature on HIPs and College Outcomes

Foundational scholarship regarding the relationship between HIP participation and

colleges outcomes has historically drawn from studies that use data from the National Study of

Student Engagement (NSSE). Initial work that considered educationally purposeful activities

which included but were not exclusive to HIPs identified multiple positive and significant albeit

relatively small effects for both first-year and graduating seniors (Kuh, Kinzie, Cruce et al.,

2007). Participation in educationally purposeful activities during the first year was found to have

a positive relationship with both first-year grade point average (GPA) and persistence to the

second year while a similar small but significant relationship was identified between

participation in educationally purposeful activities and fourth-year GPA. In addition to these

main effects, participation in educationally purposeful activities yielded evidence of conditional

effects within the first-year student cohort with positive effects related to first-year GPA for

students who entered with lower ACT and positive effects related to persistence to the second

year for African American students.

This foundational study gained scholarly prominence via a series of subsequent papers

(Kinzie et al., 2008; Kuh, Kinzie, Buckley, et al., 2007; Kuh et al., 2008) and was also featured
 71

in a 2008 AAC&U monograph entitled High Impact Practices: Who They Are, Who Has Access

to The, and Why They Matter (Kuh, 2008). This monograph also incorporated findings from the

2007 NSSE dataset that specifically examined the relationship between participation in

individual HIP experiences and self-reported student learning (National Study of Student

Engagement, 2007). Results of this analysis were mixed with findings suggesting that

participation in learning communities and service learning yielded the most substantive gains in

deep learning for the first-year cohort while an effect of similar magnitude was identified for

participation in undergraduate research for the senior year cohort. The relationships between

participation in other HIPs and deep learning were generally positive and more moderate in

magnitude.

With early findings signaling benefits of HIP participation, Kuh (2008) recommended

that undergraduate students participate in at least two HIP experiences with one situated early in

their enrollment and one sequenced closer to graduation. Using NSSE data collected from 2006-

2008, Finley and McNair (2013) identified a positive relationship between increased

participation in HIP experiences and self-reported gains in general education, practical

competence, and personal and social development. Although these patterns were reflected both

in aggregate with the full analytic sample and when considering differences across student

race/ethnicity, gains appeared to be particularly salient for first-generation students with self-

reported gains in each of the aforementioned areas exceeding those of continuing generation

students (Finley & McNair, 2013).

Extending the HIP Discourse

 72

Despite the positive effects of participation in HIP experiences identified in early studies,

the literature was not without limitation related to sample selection, methodology, and outcomes

(Brownell & Swaner, 2010). Moreover, as researchers sought to address some of these

limitations, the relationship between participation in HIP experiences and college outcomes

became more nuanced. More recent studies that use the NSSE dataset offer continued to provide

evidence that learning gains vary by HIP experience (Valentine et al., 2021) Additionally, these

gains varied not just by type but also magnitude based on student race/ethnicity (Valentine et al.,

2021; Zilvinskis, 2019). Similar findings related to differential effects in post-graduation plans

were identified by Miller et al. (2018).

As the literature base expanded to include studies with multi-institutional datasets other

than the NSSE, further evidence emerged that clarified and sometimes contradicted the

prevailing discourse. Both the Wabash National Study of Liberal Arts Education (WNS) and the

Educational Longitudinal Study (ELS) offered researchers variables from multiple waves of data

collection, thereby permitting a broader range of pre-college characteristics to be incorporated

into analytic models and more diverse sets of college outcomes to be considered. Studies using

the WNS contributed to the growing body of evidence that learning gains might vary across

individual HIP experiences (Kilgo et al., 2015; Seifert et al., 2014) and that an overemphasis on

general effects may mask interaction effects for minoritized student populations (Seifert et al.,

2014). Little consensus emerged regarding the relationship between HIP participation and

college completion with some evidence suggesting no relationship exists (Johnson & Stage,

2018) and other findings suggesting not only a positive effect but also a cumulative one

(McDaniel & VanJura, 2020). Finally, post-graduation outcomes began to be examined with
 73

positive and cumulative effects of HIP participation identified for civic engagement in early

adulthood (Myers et al., 2018) and varying effects by major for early career outcomes such as

self-reported earnings and attitudes toward current jobs (Wolniak & Engberg, 2019).

Despite the expansion of the literature base relating to HIP participation and college

outcomes, few studies have specifically accounted for the issue of student choice (Astin, 1984), a

notable omission as HIP experiences are not universally required for graduation. Observational

studies that do not account for the opt-in nature of these activities are subject to issues of

selection bias (Rosenbaum, 2002) in that students who elect to participate in a particular

experience may be different from those who choose not to participate. Propensity score (PS)

analysis offers researchers a variety of techniques including matching, weighting, and

stratification to reduce bias (Austin, 2011; Stuart, 2010; Thoemmes & Kim, 2011). While many

studies include covariates in their respective analyses, PS techniques are more advantageous in

that they permit group differences to be balanced independent of the outcome under study,

thereby reducing the potential for error (Steiner et al., 2010). Indeed, the studies that have

incorporated PS adjustments in the examination of the relationship between HIP participation

and college outcomes (Bowman & Holmes, 2018; Culver & Bowman, 2019; Provenchar &

Kessel, 2019) have generally found that the absence of such adjustments resulted in

overestimation of the effects of HIP participation.

Limitations in Previous Research

While considerable progress has been made to address some of the limitations in the

foundational HIP scholarship (Brownell & Swaner, 2010), further inquiry is warranted. The

dominance of observational studies that do not account for selection bias via PS analytic
 74

techniques or other methods in both foundational and contemporary scholarship is particularly

notable (Finley & McNair, 2013; Kilgo et al., 2014; Kuh, Kinzie, Cruce, et al., 2007; Kuh, 2008;

McDaniel & Van Jura, 2020; Miller et al., 2018; Myers, et al., 2018; Seifert et al., 2014;

Valentine et al., 2021; Wolniak & Engberg, 2019). Moreover, the studies that incorporated these

types of controls likewise exhibited some of the limitations identified by Brownell and Swaner

(2010) in that they used data from a single institution (Provenchar & Kassel, 2019) or focused on

a single HIP experience (Bowman & Holmes, 2018; Culver & Holmes, 2019). Additionally, few

studies have identified benefits of cumulative participation in HIP experiences (Finley &

McNair, 2013; McDaniel & Van Jura, 2020; Myers et al., 2018) and of those, just one has

focused on post-graduation outcomes (Myers et al., 2018).

Current Study

The present study was designed to extend existing scholarship on the relationship

between participation in two or more HIP learning experiences as recommended by Kuh (2008)

and post-graduation college outcomes using the dataset from the Educational Longitudinal Study

(ELS) of 2002. With a focus on the relationship between cumulative participation in HIP

experiences and perceptions of preparedness for adult life, the present study was guided by the

following research questions:

1. Does participation in two or more types of HIP experiences affect the perceived

importance of postsecondary experience in preparation for adult life when

controlling for student precollege sociodemographic characteristics, high school

academic preparation, organizational factors, and curricular and cocurricular

individual student experiences?

 75

2. Is the relationship between participation in two or more types of HIP experiences

and the perceived importance of postsecondary experience in preparation for adult

life moderated by student race/ethnicity, student sex, student SES, and high

school academic achievement?

The present study explored these research questions via PS analysis using structural

equation modeling (SEM), a methodological approach developed and tested by Leite et al.

(2019). Using MPlus software, this approach incorporated the ability to account for self-selection

using PS analysis as well as the complex relationships among diverse variables as represented by

contemporary college outcomes scholarship (Reason, 2009; Terenzini & Reason, 2005) using

SEM. With a multiple-group SEM design, the methodology allowed for examination of

differences between students who participated in two or more HIPs and those who did not.

Finally, the use of sensitivity analysis permitted consideration of the degree to which omitted

variables may influence the results of model testing (Harring et al., 2017).

Data Source

Sponsored by the National Center for Educational Statistics (NCES), the Educational

Longitudinal Study (ELS) served as the data source for the present study. It was designed to

examine moments of significant transition in students’ educational experiences beginning in high

school and continuing into postsecondary education as well as their careers. High school

completion, college access and persistence, and postsecondary outcomes were central foci of the

ELS along with college access and choice. The ELS aggregated information related to factors

that might influence decision making at these points of transition including student background,
 76

family support, high school experiences, postsecondary experiences, and employment

experiences as well as high school, college, and post-college outcomes (Ingels et al., 2004).

The ELS dataset was particularly suited to the present study as it contained variables that

correspond with a range of factors associated with both traditional (Astin, 1984; Astin, 1993;

Tinto, 1993; Terenzini et al., 1997) and contemporary college outcomes scholarship (Reason,

2009; Terenzini & Reason, 2005). Information pertaining to students’ precollege background

including sociodemographic characteristics, academic preparation and performance in high

school, and student dispositions was present in the dataset. Structural-demographic

organizational factors were likewise present as were curricular and cocurricular individual

student experiences. That said, other elements of particular focus in the contemporary college

outcomes discourse (Reason, 2009; Ro et al., 2013) including organizational behavior measures,

peer environment indicators such as campus racial climate and campus academic climate, and

individual classroom experiences were not included and remain topics for future research.

Of particular relevance were the ELS items related to HIP participation. A composite

variable representing cumulative student participation in a range of HIP experience types as well

as dichotomous indicators representing individual participation in internships, research, study

abroad, community-based projects, and capstone experiences were included in the ELS dataset11.

While these were indirect measures that relied solely on self-reported participation, they served

11
The HIP participation measures in the ELS were distinct among available NCES datasets. Collected in 2012, this
data was released for public use in 2014. While more recent studies including the Beginning Postsecondary Students
Longitudinal Study (BPSL) of 2012 and the HSLS: 2009 included measures of HIP participation, neither was as
comprehensive across diverse experience types as the ELS. The BPSL dataset included questions about participation
in study abroad but not internships, research, community-based projects, or capstone experiences. The HSLS:2009
dataset included only three HIP activities: research, study abroad, and community-based projects.
 77

as indicators of participation in certain types of HIPs and facilitated understanding of cumulative

participation across different types of experiences.

The ELS data source contains variables collected from diverse sources over four rounds

of data collection across 15 years (Ingels et al., 2004). Base year data collection began in 2002

which represented the sophomore year of high school for the cohort. Subsequent data collection

consisted of the first follow-up questionnaire in spring 2004 (Ingels et al., 2005), the second

follow-up questionnaire in spring 2006 (Ingels et al., 2007), and the third and final follow-up

questionnaire in summer 2012 (Ingels et al., 2014). High school transcripts were collected during

the 2004 first follow-up and postsecondary transcripts were collected during the 2012 third

follow-up.

Variables for the present study were derived from data collected during the second, third,

and fourth waves of data collection. A subset of variables was extracted from the 2004 first

follow-up questionnaire, high school transcript, the 2006 second follow-up questionnaire, and the

2012 third follow-up questionnaire. The largest percentage was variables was selected from the

third follow-up questionnaire given the focus on postsecondary experiences and post-graduation

college outcomes as shown in Appendix B.

Analytic Sample

The analytic sample for the present study consisted of the ELS twelfth grade student

cohort which was representative of the population of high school seniors in 2004 (Ingels et al.,

2014).12 Moreover, with research questions focused on post-graduation college outcomes, only

12
Since a third follow-up weight was employed in the present study per NCES guidelines, the G12 cohort should
include respondents who were designated as cohort members during the first follow-up (G12COHRT =1) as well as
those who were not designated as cohort members until the second follow-up (G12COHRT =2) (Ingels et al., 2014).
 78

students who had earned a bachelor’s degree or higher at the time of the third follow-up

questionnaire were included in the analytic sample (F3ATTAINMENT  6 (bachelor’s degree)).

This approach was consistent with existing scholarship that has previously used the ELS dataset

to examine the relationship between HIP experiences and post-graduation college outcomes

(Wolniak & Engberg, 2019). These specifications yielded a prospective analytic sample in which

37.29% of cases had data missing on at least one variable with 148 missing data patterns. Given

both the nature of the data structure and the complexity of the analytic techniques, full case

analysis was employed for the present study, yielding a final analytic sample of 3,105 cases.

Missing data is a common issue in survey datasets related to postsecondary experiences

(Cheema, 2014) and as such, researchers frequently deploy a variety of techniques to address

these challenges including expectation-maximization (EM), full-information maximum

likelihood (FIML) and multiple imputation (MI). Although EM, FIML, and MI were evaluated

for prospective use for the present study consistent with relevant scholarship (Cox et al., 2014),

the complexity of the analytic approach raised further challenges. While multiple studies have

addressed the issue of combining PS estimates generated by MI (Hill, 2004; Mitra & Reiter,

2016), there is little empirical support for procedures to combine fit indices generated by MI

(Enders & Mansolf, 2016). Additionally, the categorical nature of multiple covariates in the

model precluded the use of EM and FIML at distinct points in this analysis, ultimately resulting

in the use of full case analysis. While not the most sophisticated approach currently available to

researchers, full case analysis can be reasonably employed under the right conditions (Cox et al.,

2014) related to the nature of the missingness (Allison, 2002) and sufficiency of included

covariates (Graham, 2009).

 79

Within the analytic sample, 68.99% identified as White while 12.37% identified as Asian,

6.66% as Black or African American, 7.86% as Hispanic, 3.77% as multiracial, and 0.35% as

American Indian/Alaska Native. 58.33% identified as female with 41.67% identifying as male.

Mean SES was 0.48 with a standard deviation of .71 and a range from -2.12 to 1.97. 76.97%

earned a cumulative high school GPA of 3.0 or above with 42.54% earning a high school GPA

of 3.5 or above. 35.91% of analytic sample members planned to graduate with a bachelor’s

degree while an additional 36.94% expected to attain a master’s degree or equivalent, and

23.22% expect to earn a PhD, MD, or other advanced degree as of their senior year of high

school. Many analytic sample members (68.28%) attended a public institution as their first

postsecondary institution. A larger majority (87.12%) enrolled in a four-year institution as their

first postsecondary institution. Most completed their studies at their first postsecondary

institutions as 34.75% of respondents in the analytic sample transferred from their first

postsecondary institutions.

Outcome Measure

The present study adopted a cross-sectional design with a post-graduation latent13

outcome measure focused on students’ perception of the importance of their postsecondary

experiences for preparation for adult life (PREP). This variable was derived from ELS items:

F3A15A-F3A15D with the item stem “How important would you say your [undergraduate

education was in/college education was in/college attendance has been in] preparing you for the

following aspects of your life?” (National Center for Educational Statistics, n.d.). Collected via

13
Latent variables are measured indirectly using responses from a set of questions or test items (Schumacker &
Lomax, 2016), differing from observed variables which can be measured directly. The use of both observed and
latent variables in SEM models permits researchers to account for substantive complexity in relationships between
variables (Schumacker & Lomax, 2016; Streiner, 2006; Wang & Wang, 2012).
 80

the 2012 third follow-up questionnaire, F3A15A targeted work and career, F3A15B denoted

further education, F3A15C financial security, and F3A15D indicated civic participation

(National Center for Educational Statistics, n.d.). These items were measured on a three-point

scale: very important (1), somewhat important (2), and not at all important (3). For purposes of

this analysis, the scales were reverse coded to facilitate clarity in interpretation. For the analytic

sample, Coefficient Alpha for the PREP latent outcome measure was 0.68.

Treatment Measure

The treatment measure was a dichotomous variable that represented student participation

in two or more types of HIP experiences while enrolled as an undergraduate student at a

postsecondary institution (HIPREC). This variable was calculated using ELS items F3A14A-

F3A14E with the item stem “Have you participated/Did you participate] in any of the following

as a part of your [undergraduate/college] education?” (National Center for Educational Statistics,

n.d.). HIP experience data collected during the 2012 third follow-up included (a) internship, co-

op, field experience, student teaching, or clinical assignment (F3A14A), (b) research project with

a faculty member outside of course or program requirements (F3A14B), (c) study abroad

(F3A14C), (d) community-based project as part of a regular course (F3A14D), and (e)

culminating senior experience, such as a capstone course, senior project or thesis, or

comprehensive exam (F3A14E) (National Center for Educational Statistics, n.d.). ELS variables

F3A14A-F3A15E were dichotomous indicators with response options as follows: no (0) and yes

(1). These variables were included in the third follow-up questionnaire administered in 2012.

For the treatment measure (HIPREC) in the present study, sample members were coded

as 0 if they responded yes to fewer than two of the ELS F3A14A-F3A15E items. They were
 81

coded as a 1 if they responded yes to two or more of the ELS F3A14A-F3A15E questions.

Within the analytic sample, the group of respondents who reported two or more HIP experiences

during their undergraduate career were designated as the treatment group (1) as they completed

or exceeded the recommended number of HIP experiences (Kuh, 2008). Correspondingly, the

group of respondents who indicated that they had less than two HIP experiences during their

undergraduate career were considered to be the control group (0). 54.17% of analytic sample

members reported participation in two or more HIP experiences during their undergraduate

enrollment.

Model Covariates

The 24 covariates in this model represented a variety of domains to promote the efficacy

of PS analysis in accounting for selection bias (Steiner et al., 2015). These domains included

student precollege characteristics, organizational factors, and individual student experiences that

have been identified as influential to student learning and persistence in contemporary theoretical

frameworks related to college outcomes (Reason, 2009; Terenzini & Reason, 2005). Please see

Appendix B for a list of all covariates including a brief description of each as well as the ELS

reference variables that informed variable specification.

As outlined by Reason (2009), student sociodemographic traits such as race/ethnicity,

student sex, and student SES are important elements within the domain of precollege

characteristics. In this model, race/ethnicity (SCSD-RACE) was represented by a dichotomous

covariate in which American Indian/Alaska Native, non-Hispanic; Black or African American,

non-Hispanic; Hispanic, no race specified; Hispanic, race specified; and More than one race,

non-Hispanic were recoded as members of minoritized populations (1). Asian, Hawaii/Pacific

 82

Islander, non-Hispanic and White, non-Hispanic were coded as members of non-minoritized

populations (0). These groupings reflect evidence of higher persistence for Asian and White

students than members of other populations (Reason, 2003). Biological sex (SCSD-SEX) was

likewise a dichotomous variable with female (0) and male (1) categories. Socioeconomic status

(SCSD-SES) was a composite continuous measure which included indicators of total family

income from all sources in the base year and the highest level of education attained by the

student’s mother and father along with measures of occupational prestige for both the mother and

father as of the 2004 first follow-up (National Center for Educational Statistics, n.d.).

A less explored aspect of student precollegiate characteristics and experiences pertains to

support from family and friends for college enrollment, an area that Reason (2009) particularly

noted as warranting further exploration. The degree to which students perceive support for

postsecondary enrollment after high school was included as a latent covariate (SCSD-FAMSUP)

representing student perceptions of support from their mother, father, close relative, and friend

Individuals who selected a don’t know response option were recoded as 0 to reflect an absence of

knowledge about the aspirations of the respondent’s family and friends (Lombardi et al., 2011).

The response option noting that respondents should do what they want to do was recoded as a 1.

Support for all other non-college options including a full-time job, vocation or apprenticeship,

military service, and marriage were collapsed into a single non-college category (2). The college

enrollment response option was recoded as college enrollment (3). For the analytic sample,

Coefficient Alpha for this latent covariate was 0.70.

With indicators of academic preparation and performance among the strongest predictors

of college outcomes (ACT, 2007), the present study incorporated both an observed covariate
 83

representing academic performance and a latent covariate that reflects students’ perception of the

degree to which their high school experiences prepared them for postsecondary education. With

evidence signaling a positive and consistent relationship between high school GPA and college

graduation (Adelman, 2006; Allensworth and Clark, 2020), the SCHS-GPA measure served as

an indicator of student’s prior academic achievement, capturing cumulative GPA from ninth

grade to twelfth grade. The SCHS-PREP latent covariate measured respondent perceptions of the

degree to which their high school classes in math, science, and English/writing prepared them for

enrollment at their first postsecondary institution. For the analytic sample, Coefficient Alpha for

this latent covariate was 0.52.

Given prior scholarship documenting the positive relationship between academic goals

and college persistence (Brown et al., 2008; Robbins et al., 2004), the SCD-GOAL measure

served as a dispositional indicator of student’s educational goals as of their senior year in high

school in 2004. Respondents indicated the highest level of education that they expected to

achieve: less than high school graduation (1), GED or other equivalency only (2), high school

graduation only (3), attend or complete 2-year college/school (4), attend college, 4-year degree

incomplete (5), graduate from college (6), obtain master’s degree or equivalent (7), obtain PhD,

MD, or other advanced degree (8), and don't know (9). For purposes of this analysis, the don’t

know (9) response option was recoded as a 0, reflecting an absence of certainty related to

educational goals at the time of measure (Lombardi et al., 2011).

Organizational context covariates were derived from the structural-demographic

characteristics of the first postsecondary institution attended by the student respondent. The

inclusion of these covariates was intended to reflect the significant influence of the first
 84

institution on educational attainment (Pascarella & Terenzini, 2005). Institutional control

(ORG1-CON) was a dichotomous indicator representing public (0) and private (1) institutions.

Similarly, institutional level (ORG1-LEV) was a dichotomous indicator representing two-year

institutions (0) and four-year institutions (1).

The individual student experience domain for the present study necessarily included

curricular covariates that reflected student enrollment and field of study. Consistent with recent

scholarship that identified interaction effects of HIP participation on post-graduation college

outcomes that varied by student major (Wolniak & Engberg, 2019), six dichotomous covariates

served as indicators of the academic field of study for the student’s first known bachelor’s degree

(see Appendix C). Additionally, transfer status relative to first postsecondary institution

(EXPCU-TSFR) was included as a dichotomous covariate indicating transfer status: no

credential earned from and no longer attending first postsecondary institution (1) and credential

earned or still attending first postsecondary institution (0).

The final domain of individual student experiences included covariates that reflect

experiences that occurred outside the classroom. Cocurricular experiences during the first two

years of enrollment included faculty interaction outside of class meetings (EXPCO-FAC),

academic advisor engagement (EXPCO-ADV), participation in intramural or nonvarsity sports

(EXPCO-INT), participation in varsity or intercollegiate sports (EXPCO-VAR), and

participation in extracurricular activities (EXPCO-EXT). Employment was captured as two

separate continuous covariates noting the number of jobs held during the first (EXPCOJOB1)

and second (EXPCOJOB2) years of enrollment. Since lack of a mentor has been identified as a

barrier to HIP participation (Finley & McNair, 2013), a variable representing access to a mentor
 85

(EXPCO-MEN) was also included as a dichotomous indicator indicating access (1) or lack

thereof (0).

Weights

In order to account for the complexities of the longitudinal design including unequal

patterns of selection and nonparticipation from selected sample members (Ingels et al., 2014), the

F3FITSCWT sampling weight was applied in alignment with NCES guidelines for analyses with

the G12COHORT that includes variables from the first follow-up questionnaire, the high school

transcript data collection, the second follow-up questionnaire and the third follow-up

questionnaire (Ingels et al., 2014). Additionally, balanced repeated replication (BRR) weight

adjustments were applied to facilitate accurate estimations of standard errors (McNeish et al.,

2017; Stapleton, 2008). In order to support these adjustments, six sets of 200 BRR weights were

computed by NCES consistent with the agency’s guidelines (Ingels et al., 2014). This guidance

further specified that F3F1T001-F3F1T200 BRR set corresponds with the selected F3FITSCWT

sampling weight.

Methods

The present study employed propensity score analysis (PS) analysis with structural

equation modeling (SEM) using MPlus software (Muthén & Muthén, 2017) following the

analytic approach proposed and tested by Leite et al. (2019). Due to the categorical nature of

many of the covariates in the model, models were fit with weighted least squares with adjusted

means and variances (WLSMV) estimation (Beauducel & Herzberg, 2006). They were run with

and without replication weights. While the models were evaluated for exact fit using a modified

Chi-Square (χ2) (Stapleton, 2008), conventional practice in SEM stipulates the use of multiple
 86

indices to assess fit (Schumacker & Lomax, 2016). Fit indices recommended by Leite et al.

(2019) for these purposes included the Comparative Fit Index (CFI), the Tucker-Lewis Index

(TLI), and the Root-Mean-Square Error of Approximation (RMSEA). For all steps of this

analysis, fit was determined to be close if CFI  0.95, TLI  0.95, and RMSEA  0.05.

Step 1: Multiple-Group Confirmatory Factor Analysis (CFA) for Latent Covariates

Following the data preparation, a multiple-group CFA was fit in MPlus (Muthén &

Muthén, 2017) with the two latent covariates (SCSD-FAMSUP and SCHS-PREP) and the

treatment and control groups to verify the factor structure of the latent covariates. Additionally,

measurement equivalence of the latent covariates across treatment and control groups was

evaluated. Known as scalar invariance (Millsap & Olivera-Aguilar, 2012), this condition allows

researchers to appropriately compare means and variances across the treatment and control

groups and is a required for mitigating selection bias in PS analysis (Leite et al., 2019). In

MPlus, scalar invariance is tested with an adjusted likelihood ratio test (Asparouhov et al., 2006).

In this case, Chi-Square (χ2) served as the relevant test statistic with an absence of statistical

significance indicating that the assumption of scalar invariance has been met.

Step 2: SEM for PS Estimation

Once the factor structure and scalar invariance were confirmed for the latent variables,

the SEM depicted in Appendix D was fit in MPlus (Muthén & Muthén, 2017) to calculate PS

weights. This model was hypothesized to represent variables that have been demonstrated to

correlate with participation in HIP experiences in the college outcomes literature. The inclusion

of these covariates was designed to mitigate the issue of selection bias associated with the opt-in

nature of HIP experiences (Steiner et al., 2010).

 87

The SEM was run with the latent and observed covariates as predictors and the HIPREC

treatment measure as the outcome. Given that the literature suggests a cumulative benefit of HIP

participation to participants in increasing numbers of HIP experiences (Finley & McNair, 2013;

McDaniel & Van Jura, 2020; Myers et al., 2018), PS weights were estimated using average

treatment effect on the treated (ATT) weights (Heckman, 2005). The PS weights for the

treatment and control groups were then compared to evaluate the assumption of overlap in PS

analysis. For this analysis, the assumption of overlap was determined to have been met if neither

group had a PS with a value of 0 or 1 (Leite et al., 2019).

Step 3: Latent Covariate Balance Evaluation and Step 4: Observed Covariate Balance

Evaluation

Following the calculation of the PS weights, these values were multiplied by the

F3FITSCWT sample weight to obtain final PS weights. Covariate balance was evaluated using

regression models of the latent and observed covariates on the treatment measure. This

assessment permitted the researcher to compare the distributions of the treatment and control

groups following application of the final PS weights via examination of the standardized mean

differences across the regression coefficients (Leite et al., 2019). This assessment was performed

using two sets of criteria given a lack of consensus in the literature. Austin (2011) stipulates

absolute values of standard mean differences below 0.100 standard deviations as evidence of

adequate covariate balance while Stuart and Rubin (2007) designate .250 as an appropriate

threshold by which to evaluate covariate balance.

Step 5: Multiple-Group CFA for Latent Outcome Using PS Weights

 88

Following the estimation of the PS weights, a multiple-group CFA was fit in MPlus

(Muthén & Muthén, 2017) for the outcome measures using the final PS weights with the sample

weight adjustments. These were run with the adjusted PREP latent outcome variables and the

treatment and control groups. This CFA was used to affirm the PREP factor structure and

evaluate scalar invariance (Millsap & Olivera-Aguilar, 2012) of the outcome measure across the

treatment and control groups using the adjusted likelihood ratio test (Muthén & Muthén, 2017).

As with the previous scalar invariance analysis, Chi-Square (χ2) served as the relevant test

statistic with an absence of statistical significance indicating that the assumption of scalar

invariance was met.

Step 6: Multiple-Group CFA for ATT Estimation Using PS Weights

In addition to testing for scalar invariance, the multiple-group CFA was used to estimate

the ATT using PS weights. As noted in Step 5, this model was run with the adjusted PREP

variable and the treatment and control groups. The mean of the outcome measure for respondents

with no HIP experiences was set to zero to identify the model. With this adjustment, the resultant

mean of the outcome measures for respondents who have participated in two or more HIP

experience represented the latent mean difference between the groups, i.e., the ATT (Leite et al.,

2019).

Step 7: Multiple-Group SEM for Interaction Effect Estimation

A multiple-group SEM was used to evaluate interactions (Wang & Wang, 2012) between

the outcome measure and the following observed covariates: race/ethnicity (SCSD-RACE), sex

(SCSD-SEX), SES (SCSD-SES), and high school GPA (SCHS-GPA). In this model,

participation in two or more HIP experiences (HIPREC) served as a grouping variable with
 89

sample members who completed two or more HIPs considered to be the treatment group and

those who completed fewer than two HIPs designated as the control group (see Appendix E).

The estimated differences for the coefficients between the groups were used to determine the

nature of any identified interactions. Although no covariates were needed due to the presence of

the final PS weights, linear, cubic, and quadratic PS estimates were added to achieve doubly

robust estimation and account for any lingering bias in the model (Bang & Robins, 2005; Schafer

& Kang, 2008).

Step 8: Multiple-Group SEM with Phantom Variable for Sensitivity Analysis

Using the PS weights, a multiple-group SEM with a phantom variable was fit in MPlus

(Muthén & Muthén, 2017) to evaluate in the sensitivity of the model to omitted confounders

using the methodology specified by Harring et al. (2017). Consistent with Leite et al. (2019),

nine multiple-group SEMs were used with varying permutations of sensitivity parameters. The

first parameter which corresponded with SMD on the phantom variable was tested with

parameters of 0.25, 0.5, and 1.0. It was tested in combination with the second parameter which

consisted of the standardized path from the phantom variable to the outcome measure, likewise

with parameters of 0.25, 0.5, and 1.0. If the p values for any of the interactions across the

multiple-group SEMs demonstrated a change in statistical significance (i.e., from non-

significance to statistical significance) following the parameter adjustment, it was interpreted to

signal the absence of potential confounders.

Results

The multiple-group CFA provided evidence to support the factor structure of the latent

covariates SCSD-FAMSUP and SCSD-HSPREP. As shown in Table 1, the fit indices signal
 90

close, albeit not exact, fit under conditions of both configural and scalar invariance. The adjusted

likelihood ratio test (Asparouhov et al., 2006) that served as a comparison between configural

and scalar invariance was not statistically significant (χ2 (14) = 13.660, p = .475). The lack of

statistical significance indicated that the assumption of scalar invariance was met, thereby

permitting comparison of means and variances across the treatment and control groups.

Table 1

Manuscript I: Model Fit Statistics and Indices

Estimation of the PS scores with the SEM presented in Appendix D likewise yielded

model fit that was close but not exact as shown in Table 1. Analysis of the descriptive statistics

for the PS scores further indicated that the requisite assumption of overlap was met with neither

group containing a PS score of zero or one. For students who participated in two or more HIPs,

minimum, maximum, and mean propensity scores were .117, .965, and .636. Respective analogs

for students who did not participate in two or more HIPs were .124, .939, and .477.

Following PS estimation, PS weights for the ATT were calculated by multiplying the PS

estimates by the F3FITSCWT sample weight. Inspection of the PS weights for the ATT yielded a

maximum value of 15.393. Since this value did not exceed the sample size, it was assumed that

extreme weights were unlikely to impact subsequent analyses (Leite et al., 2019). As shown in

Table 2, the results of the covariate balance evaluation for both the latent and observed

covariates mostly demonstrated adequate ATT-weighted balance with absolute values for the
 91

standardized means differences equal to or less than .100 for all but two covariates (i.e.,

EXPCUEDU and EXPCUHHS) (Austin, 2011) and less than .250 for all model covariates

(Stuart & Rubin, 2007).

Table 2

Manuscript I: Standardized Mean Differences (SMD) for Covariate Balance Evaluation

Covariate ELS Source Description Baseline ATT

Variable a SMD weighted
SMD
Student Precollege Characteristics and Experiences
Sociodemographic traits
SCSD-RACE F1RACE Membership in a minoritized racial population -.119 .015
SCSD-SEX F1SEX Biological sex -.173 -.034
SCSD-SES F1SES2 SES .194 -.013
SCSD-FAMSUP F1S44A-F1S44D Perception of support network aspirations .053 .031

Academic performance and preparation

SCHS-GPA F1RGPP2 High school GPA .237 -.008
SCHS-PREP F2B17A-F2B1C Perception of academic preparedness .187 -.031

Student dispositions
SCD-GOAL F1STEXP Postsecondary educational expectations .140 -.001

Organizational Context
ORG1-CON F3PS1SEC Control of first postsecondary institution .474 -.014
ORG1-LEV F3PS1SEC Level of first postsecondary institution .366 .034

Individual Student Experiences

Curricular experiences
EXPCU-TSFR F3PS1RETAIN Student respondent’s transfer status -.185 -.075

Field of study for first known bachelor’s degree

EXPCU-ART F3TZBCH1CIP2 Arts and humanities .046 .001
EXPCU-BUS F3TZBCH1CIP2 Business -.467 -.001
EXPCU-ED F3TZBCH1CIP2 Education .097 .129
EXPCU-HHS F3TZBCH1CIP2 Health and human services .202 -.121
EXPCU-SS F3TZBCH1CIP2 Social science .117 .093
EXPCU-STEM F3TZBCH1CIP2 Science, technology, engineering, and math .113 -.058

Out-of-class experiences
EXPCO-FAC F2B18A Talk with faculty outside of class .312 .006
EXPCO-ADV F2B18B Meet with their advisor about academic plans .134 .061
EXPCO-INT F2B18E Participate in intramural or nonvarsity sports .140 .004
EXPCO-VAR F2B18F Participate in varsity or intercollegiate sports -.016 -.058
EXPCO-EXT F2B18G Participate in extracurricular activities .437 -.062
EXPCO-JOB1 F2C24_P Number of jobs while enrolled in 2004-2005 -.026 -.024
EXPCO-JOB2 F2C29_P Number of jobs while enrolled in 2005-2006 .035 .023
 92

EXPCO-MEN F3A14F Access to mentoring experience .802 -.052

a
The first two characters of the ELS source signal the corresponding data collection wave: F1 variables were
collected in 2004, F2 variables in 2006, and F3 variables in 2012.

Once the adequacy of covariate balance was established, a multiple-group CFA was

conducted to verify the factor structure of the PREP latent outcome variable. As shown in Table

1, the fit indices provide evidence of close but not exact fit under conditions of both configural

and scalar invariance. With a non-significant result for the adjusted likelihood ratio test

comparing configural and scalar invariance (χ2 (6) = 6.969, p = .324), the assumption of scalar

invariance of the adjusted PREP latent outcome variable was determined to have been met

(Asparouhov et al., 2006).

The multiple-group CFA with scalar invariance was run with the adjusted latent outcome

variable and the treatment and control groups to estimate the ATT using PS weights. With the

mean of the outcome measure for respondents without the recommended number of HIP

experiences set to zero, the mean of the outcome measures for respondents who participated in

two or more HIP experiences represented the ATT estimate (Leite et al., 2019). The ATT

estimate of the perceived importance of postsecondary experience in preparation for adult life

(PREP) was statistically significantly different from zero without BRR weights ( = .182, SE =

.088, p = .039). However, when BRR weights were used to improve the accuracy of the standard

error (SE) estimation, the corresponding ATT estimate was not statistically significant ( = .182,

SE = .095, p = .055).

Following the estimation of the multiple-group CFA, a multiple-group SEM to estimate

the interaction effects under study was run. Linear, quadratic, and cubic PS estimates were added

to the model for a doubly robust ATT estimation (Bang & Robins, 2005; Schafer & Kang, 2008).
 93

As shown in Table 1, this model demonstrated close but not exact fit. The doubly robust ATT

estimate of the perceived importance of postsecondary experience in preparation for adult life

(PREP) was not statistically significantly different from zero without BRR weights ( = .806, SE

= .825, p = .328). However, when BRR weights were added, the corresponding ATT estimate

was statistically significant ( = .806, SE = .129, p = .000).

Additionally, the results of multiple-group SEM also revealed several interactions as

shown in Table 3. Group differences that were statistically significant served as evidence of an

interaction. There was a negative relationship between membership in a minoritized racial group

(SCSD-RACE) and perceived importance of postsecondary experience in preparation for adult

life (PREP) for students who did not participate in the recommended number of HIPs. Control

group respondents who were members of minoritized racial groups (coded as 1) reported lower

perceived importance of postsecondary experience in preparation for adult life than respondents

who were not members of minoritized racial groups (coded as 0). However, there was a positive

relationship between membership in a minoritized racial group and PREP for students who

participated in two or more HIPs, a difference that was statistically significantly different from

zero (p = .004). Thus, respondents who were members of minoritized racial groups and

participated in two or more HIP experiences reported higher perceived importance of

postsecondary experience in preparation for adult life than respondents who were not members

of minoritized racial groups.

Table 3

Manuscript I: Interaction Estimates from Multiple-Group SEM

Path Coefficients with ATT weights

Group No HIPREC (SE) HIPREC (SE) Difference (SE)
SCSD-RACE -.021 (.043) .095* (.026) -.180* (.063)
 94

SCSD-SEX -.085* (.023) -.220* (.015) .255* (.027)

SCSD-SES -.079* (.018) -.119* (.014) .097* (.021)
SCHS-GPA .052* (.008) .033* (.008) .009 (.016)
Propensity score 1.093* (.153) .443* (.055)
Propensity score squared -2.634* (.335) -.363* (.048)
Propensity score cubed 2.403* (.244) .457* (.058)
* Denotes statistically significant coefficients at  = .05

Additionally, as shown in Table 3, a statistically significant interaction (p = .000) was

also identified for the dichotomous biological sex variable (SCSD-SEX). There was a negative

relationship between biological sex and perceived importance of postsecondary experience in

preparation for adult life (PREP) for students who did not participate in the recommended

number of HIPs. In this analysis, male respondents (coded as 1) who participated in fewer than

two HIPs reported lower perceived importance of postsecondary experience in preparation for

adult life than female respondents (coded as 0). For students who participated in two or more

HIPs, the relationship remained negative but increased in magnitude. Thus, male respondents

who participated in two or more HIP experiences reported lower perceived importance of

postsecondary experience in preparation for adult life than female respondents.

For socioeconomic status (SCSD-SES), there was a negative relationship between the

socioeconomic status and perceived importance of postsecondary experience in preparation for

adult life (PREP) as indicated in Table 3. In this model, socioeconomic status was measured on a

continuous scale ranging from a minimum of -2 to a maximum of 2. For students who

participated in fewer than two HIPs, the negative relationship signals that as socioeconomic

status increased, perceived importance of postsecondary experience in preparation for adult life

(PREP) decreased. The relationship remained negative but increased in magnitude for students

who participated in two or more HIPs. This difference was statistically significantly different

from zero (p = .000).

 95

For high school GPA (SCHS-GPA), there was a small, albeit positive, relationship

between high school GPA and perceived importance of postsecondary experience in preparation

for adult life (PREP) as shown in Table 3. Thus, as high school GPA status increased, perceived

importance of postsecondary experience in preparation for adult life (PREP) likewise increased.

This positive relationship was present for students who did not participate in the recommended

number of HIPs. It remained positive but decreased in magnitude for students who participated

in two or more HIPs. However, this difference was not statistically significantly different from

zero (p = .563).

As shown in Table 3, linear, quadratic, and cubic PS estimates were added to the

multiple-group SEM. The inclusion of these scores permitted a doubly robust estimation to

account for any remaining bias after the weights were incorporated into the model (Bang &

Robins, 2005; Schafer & Kang, 2008). For both groups of students, linear relationships between

the PS estimates and perceived value of postsecondary experience in preparation for adult life

(PREP) were positive and statistically significant (p = .000) while the quadratic relationships

were negative and likewise statistically significant (p = .000). Cubic relationships were positive

and statistically significant (p = .000). This signaled that the relationship between probability of

participation in the recommended number of HIPs and perceived value of postsecondary

experience in preparation for adult life (PREP) is s-shaped.

Finally, results of the sensitivity analyses are displayed in Table 4. The purpose of this

analysis was to evaluate whether an unspecified omitted covariate would result in change of p-

values for the ATT and interactions across a range of sensitivity parameters. As noted, the ATT

and interactions (i.e., SCSD-RACE, SCSD-SEX, and SCSD-SES) that were statistically
 96

significant for the multiple-group SEM to estimate the interactions remained statistically

significant across all parameters sets. Likewise, the interaction for SCHS-GPA that was not

statistically significant remained so. Thus, none of the conclusions regarding the ATT and

interaction effects demonstrated evidence of sensitivity to parameter changes, thereby signaling

that the ATT and interaction effects were not influenced by omitted covariates.

Table 4

Manuscript I: P-Values from Sensitivity Analysis

Sensitivity Parameter Pairs

.1, .1 .25, .1 .5, .1 .1, .25 .25, .25 .5, .25 .1, .5 .25, .5 .5, .5
ATT .001* .000* .000* .000* .000* .001* .000* .000* .000*
SCSD-RACE .042* .005* .009* .023* .025* .032* .009* .007* .002*
SCSD-SEX .000* .000* .000* .000* .000* .000* .000* .000* .000*
SCSD-SES .000* .001* .000* .000* .001* .001* .003* .004* .000*
SCHS-GPA .334 .794 .769 .474 .937 .776 .517 .769 .496
* Denotes statistically significant coefficients at  = .05

Analytical Limitations

Despite its many advantages, several characteristics of the ELS data resulted in analytical

limitations for the present study. While the ELS contained variables that specified participation

in five types of HIP experiences, others were omitted from consideration including first-year

seminars and experiences, common intellectual experiences, learning communities, writing-

intensive courses, and collaborative assignments and projects. Additionally, there were no

variables that specified the number of individual HIP engagements, duration, or context, a

limitation that has been acknowledged in other studies (Myers et al., 2017). Similarly, there were

no indicators pertaining to the quality of those experiences, an area of increasing scholarly

interest within the HIP discourse (Zilvinskis, 2019).

Additionally, the aggregation of the distinct racial/ethnic categories in the SCSD-RACE

variable resulted in limitations to the conclusions that can be drawn in the present study (McNair
 97

et al., 2020). Aggregating these categories particularly limited understanding of the substantive

diversity across student experiences and college outcomes within minoritized student populations

in the U.S. (Patton et al., 2015). However, the racial/ethnic diversity in the analytic sample was

insufficient to permit estimation of the PS scores for each of the individual racial/ethnic

categories.

Finally, several key influences identified within the contemporary college outcomes

scholarship (Reason, 2009) were simply not measured in any of the ELS waves of data

collection. Most notably, while structural-demographic features of postsecondary institutions

were included in the ELS dataset, no variables that measured or approximated the organizational

behavior dimensions of those institutions were incorporated into the ELS dataset. Similarly,

despite the presence of variables that targeted individual student experiences in the curriculum

and co-curriculum, measurement for both was limited to the first two years of enrollment.

Variables related to classroom experiences as well as student peer environments including

campus racial climate and campus academic climate were also absent from the ELS dataset.

Discussion

Despite these limitations, the present study extends the HIP discourse with a

methodology designed to further refine understanding of the impact of cumulative participation

in HIP experiences. The ability to account for the particular issue of selection bias via PS

estimation is important for discerning whether participation in multiple types of HIP experiences

does indeed contribute to a particular outcome or whether such influences are best explained by

differences among individuals who opt to participate versus those who do not, an issue of interest

to both individual students and postsecondary institutions. The doubly robust estimation
 98

employed in this analysis is designed to address any lingering bias that is not otherwise

accounted for by the PS weights (Bang & Robins, 2005; Schafer & Kang, 2008), further

extending the established methodological discourse around PS analysis and HIP participation

(Bowman & Holmes, 2018; Culver & Bowman, 2019; Provenchar & Kessel, 2019) and

providing postsecondary educational researchers with an additional set of rigorous statistical

tools for consideration as they look to evaluate the effects of participation in opt-in learning

experiences such as HIPs.

Moreover, the present study contributes to the scholarship around the range of outcomes

that are examined in the HIP discourse. The foundational scholarship related to the effects of

participation in HIPs was dominated by studies focused on a relatively limited set of outcomes

that were primarily of interest to postsecondary institutions such as persistence, academic

achievement as defined by grade point average (GPA), and graduation rates (Kinzie et al., 2009;

Kuh, 2008; Kuh, Kinzie, Buckley, et al., 2007; Kuh, Kinzie, Cruce, et al., 2007; Kuh et al., 2008)

or on indirect, self-reported measures of student learning (Kuh, 2008; National Study of Student

Engagement, 2007). As more recent scholarship has examined direct measures of student

learning in college as well as postsecondary employment outcomes, the notion of what

constitutes impact has broadened and evidence that suggests that the nature and duration of that

impact may vary has emerged (Myers et al., 2018; Seifert et al., 2014; Kilgo et al., 2015;

Wolniak & Enberg, 2019). The use of SEM in the present study to consider latent outcome

variables such as attitudinal measures of preparation for adult life further expands the range and

complexity of outcomes that can be examined. In era of considerable uncertainty for higher

education (Friga, 2021) when the perceived value of college is being increasingly called into
 99

question (Klebs et al., 2021), the ability to explore and examine a range of attitudinal, cognitive,

and behavioral outcomes provides an opportunity to expand the scope of the value proposition

related to participation in HIP experiences beyond measures that are bounded by a student’s

undergraduate enrollment, particularly given the statistically significant main effect identified in

the present study with doubly robust estimation.

Results of this analysis further suggest variation in across demographic groups that are of

practical value for practitioners, educators, and administrators who design, implement and fund

HIP experiences. The results suggest that sample members who have historically enjoyed

advantages in terms of postsecondary access and outcomes did not perceive that postsecondary

experience was important in their preparation for adult life. Rather, male sample members as

well as those whose families reported higher SES reported lower perceived importance of

postsecondary education in preparation for adult life. This negative relationship was present both

for those who participated in two or more HIPs and those who did not. In this regard,

participation in more than two HIPs deepened the negative relationship, thereby signaling that

participation in two or more HIP experiences may have little value in terms of perceived

importance of postsecondary education in preparation for adult life. However, as Finley and

McNair (2013) acknowledge, students who have traditionally enjoyed advantages in

postsecondary access and outcomes tend to participate in HIPs at higher rates than those who

have historically been disadvantaged. That said, further investigation is warranted, particularly

given prior scholarship that suggests that benefits from HIP participation may vary based on type

of HIP experience (Miller et al., 2018; National Study of Student Engagement, 2007) and that

certain HIP experiences might be more beneficial for particular majors than others (Wolniak &
 100

Engberg, 2019). Thus, while participation in any two HIPs might be insufficient to enhance

perceived importance of postsecondary education in preparation for adult life, a particular set of

HIPs might have particular impact for students in a specific major that would otherwise absent.

Thus, the notion of which HIPs, how many, under what circumstances, and in what order should

be examined further to better understand how these choices might interact in influencing impact

for different students.

Conversely, the negative relationship presented in the interaction effects under study in

the multiple-group SEM suggested that sample members who have not historically enjoyed

advantages in terms of postsecondary access and outcomes perceived higher importance of

postsecondary education in preparation for adult life. Sample members who were women or who

had families with lower SES reported higher perceived importance of postsecondary education in

preparation for adult life. Within this sample, this positive relationship was enhanced by

participation in two or more HIPs, a finding that is consistent with earlier studies that point to

benefits for populations that have been underrepresented in postsecondary educational

enrollment (Finley & McNair, 2013; Kuh, Kinzie, Cruce, et al., 2007; Seifert et al., 2014).

Moreover, the interaction effect identified for membership in a minoritized racial/ethnic

population is of particular interest and provides evidence of a compensatory effect first identified

by Kuh, Kinzie, Cruce, et al. (2007). Sample members who were members of minoritized

racial/ethnic groups and participated in fewer than two HIPs reported lower perceived

importance of postsecondary education in preparation for adult life but those who participated in

two or more HIPs report higher perceived importance of postsecondary education in preparation

for adult life. These findings may carry particular relevance for educators who work in programs
 101

and departments that specifically support minoritized student populations as well as for

administrative departments or academic units that offer HIP experiences (e.g., career and

professional development centers, study abroad programs, undergraduate research offices). The

evidence that historically disadvantaged and minoritized students who participate in two or more

HIPs perceive higher importance of postsecondary education in preparation for adult life

suggests that these practitioners should be foregrounding this messaging in student outreach and

recruitment and helping connect students with these types of learning experiences while they are

enrolled, particularly given the evidence that suggests that they tend to participate in HIPs at

lower rates (Finley & McNair, 2013). However, despite the promising nature of these findings,

additional investigation regarding the nature of these relationships is warranted, particularly

given findings that suggest that participation in some HIP experiences may have more positive

benefits while other HIPs might yield less positive effects (Valentine et al., 2021). Moreover,

disaggregation across the race/ethnicity categories to better understand variation in specific

student experiences is needed.

Ultimately, more work remains to be done to better understand what constitutes impact

in HIPs, for whom, and under what circumstances. When such inquiry is undertaken using

statistical methods that account for factors such as selection bias, diversity in the student

population, and range of choices across the undergraduate experience while examining outcomes

that target student learning, the prospect of uncovering a nuanced understanding of the impact of

participation in HIPs is enhanced as is the quality of actionable insights that can inform decision

making for current and future college students as well as the postsecondary institutions at which

they enroll.
 102

CHAPTER V

MANUSCRIPT II

Abstract

This study examines the relationship between cumulative participation in high-impact

practices (HIP) and perceived learning and challenge in professional experiences using the

Educational Longitudinal Study of 2002 (ELS) dataset. Using a methodology proposed and

tested by Leite et al., (2019), this analysis incorporated the ability to account for self-selection

into HIPs using propensity score (PS) analysis with a multiple-group structural equation model

(SEM) design to examine differences between students who participated in two or more HIPs

and those who did not (n = 3,037). Results suggest that participation in two or more HIP

experiences was insufficient to influence perceived learning and challenge in professional

experiences as no statistically significant main effect was identified. Similarly, there were no

statistically significant differences in perceived learning and challenge in professional

experiences identified across groups who participated in two or more HIP experiences and those

who did not. Although these findings may signal that perceived benefits to learning related to

participation in HIPs may not transfer beyond postsecondary enrollment, results of the sensitivity

analysis suggested that confounding variables may be absent from the model, thereby suggesting

that future research is warranted.

Introduction

Since 2007, the scholarly discourse on college outcomes has foregrounded a set of

undergraduate student experiences that have been identified as high-impact practices (HIPs)

(Kuh, 2008). This designation refers to a set of undergraduate experiences that vary in scope and
 103

implementation: first-year seminars and experiences, common intellectual experiences (i.e., core

curriculum), learning communities, writing-intensive courses, collaborative assignments and

projects, undergraduate research, diversity and global learning, service learning and community-

based learning, internships, and capstone courses and projects (American Association of

Colleges and Universities, 2007; Kuh, 2008). Despite this breadth, these experiences share

fundamental characteristics designed to enhance student learning and college outcomes by

promoting sustained independent inquiry, facilitating student interaction with faculty and peers,

and providing opportunities for regular feedback, synthesis, and perspective taking (Kuh, 2008).

Aligned with the aspirations of Liberal Education and America’s Promise initiative

(2007) sponsored by the Association of American Colleges and Universities (AAC&U), the

earliest inquiry into HIP participation began with the examination of educationally purposeful

activities, a set of postsecondary experiences that included but was not exclusive to HIPs (Kinzie

et al., 2008; Kuh, Kinzie, Buckley, et al., 2007; Kuh, Kinzie, Cruce, et al., 2007; Kuh, 2008; Kuh

et al., 2008). Small but significant impacts from participation in educationally purposeful

activities on college outcomes including academic achievement and persistence from the first

year of enrollment to the second year were identified (Kinzie et al., 2008; Kuh, Kinzie, Buckley,

et al., 2007; Kuh, Kinzie, Cruce, et al., 2007; Kuh, 2008; Kuh et al., 2008). While

acknowledging that the inherent limitations of postsecondary institutions’ ability to improve

college outcomes for all students, this early scholarship signaled particular benefits to HIP

participation for students who come to these institutions with years of educational disadvantage

(Kuh, 2008).
 104

While foundational scholarship prioritized the examination of outcomes related to

persistence, academic achievement as defined by grade point average (GPA), and graduation

rates, subsequent studies have examined a broader range of outcomes with a subset of these

inquiries targeted to student learning and skill development. When well-designed and

thoughtfully implemented, HIP experiences share distinctive attributes related to independence,

collaboration, synthesis, and perspective taking that are inherently beneficial students (Kuh &

Kinzie, 2018) and of particular value to employers who hire those students once they graduate

(Finley, 2021; Hart Research Associates, 2015; Hart Research Associates, 2018). Moreover, the

ability to apply knowledge in real-world settings is not only reported to be valuable to employers

but participation in experiences like HIPs that provide students with opportunities to practice this

type of applied learning are also likely to influence hiring decisions (Finley, 2021). However,

while employers perceive benefit to participation in HIPs, little scholarship interrogating the

effects of postsecondary experiences on career-related attitudes aside from post-graduation job

satisfaction has been undertaken (Wolniak & Engberg, 2019).

Given these gaps, it is critical that nuances and limitations in the conclusions that can be

drawn from the literature regarding HIP participation be fully explored to ascertain more

precisely who benefits from HIP experiences, to what extent, and under what conditions those

impacts are more likely to realized. This critical examination is intended to support the ability of

postsecondary institutions to support student learning and career preparation efficiently and

effectively in a manner that maximizes the transformative potential of student engagement. A

chronological examination of the HIP scholarship offers valuable insight into how datasets,

models, and methods have evolved over time, resulting in increased complexity in terms of
 105

research design, data analysis, and actionable insights afforded by the results while also

illuminating gaps in the current scholarship that warrant further study.

Theoretical Framework

The HIP discourse has drawn heavily on traditional models of college outcomes and

student success (Astin, 1984; Astin, 1993; Tinto, 1993; Terenzini et al., 1996). With an emphasis

on the choices that students make regarding how to direct their time and energy during their

respective undergraduate enrollments (Astin, 1984), the interplay between precollege

characteristics, postsecondary experience, and postgraduation outcomes has long been

acknowledged. Astin’s Inputs-Environment-Outputs (I-E-O) model (1993) conceptualized the

inputs as student characteristics at time of entry; the environment as the people, programs, and

experiences with which the student engages as an undergraduate; and the outputs as the student

characteristics at time of departure. Tinto’s Student Departure Model (1993) offered a similar

conceptualization of this synergy. In this model, pre-entry attributes, intended and actualized

student goals and commitments, and academic and social experiences while enrolled served as a

framework for student retention. Terenzini et al. (1996) further distilled these constructs, noting

that six primary elements are central to understanding college outcomes. These elements

included precollegiate traits, curricular patterns, in-class experiences, out-of-class experiences,

institutional context, and learning outcomes.

More recently, Reason (2009) offered a more expansive view of college outcomes with a

particular focus on student persistence. Drawing on a collaboration between Terenzini and

Reason (2005), this framework foregrounded the importance of precollege characteristics and

experiences that include sociodemographic characteristics, student dispositions such as goals and
 106

motivation, and academic preparation in high school. Reason’s model (2009) also included

family support albeit with the acknowledgement that this domain represents an aspect of the

literature that is underexplored. The importance of the organizational context of the

postsecondary institution is reflected by the inclusion of structural-demographic characteristics

such as institutional control, size, and selectivity with a particular emphasis on first institution

attended consistent with the findings of Pascarella and Terenzini (2005). However, Reason

(2009) cautioned against limiting an examination of institutional context to structural-

demographic characteristics, noting that organizational behavior dimensions including

institutional priorities, campus policies, and decision-making structures are equally important, a

consideration affirmed by subsequent scholarship (Ro et al., 2013). Finally, Reason (2009)

offered important distinctions across student experiences, separating the peer environment

including the racial and academic climate from individual student experiences which can occur

in the classroom; in the curriculum via student major and participation in HIPs; and outside the

classroom via participation in programs and activities such as athletics and student organizations.

Literature on HIPs and College Outcomes

Foundational scholarship regarding the relationship between HIP participation and

colleges outcomes has historically drawn from studies that use data from the National Study of

Student Engagement (NSSE). Initial work that considered educationally purposeful activities

which included but were not exclusive to HIPs identified multiple positive and significant albeit

relatively small effects for both first-year and graduating seniors (Kuh, Kinzie, Cruce et al.,

2007). Participation in educationally purposeful activities during the first year was found to have

a positive relationship with both first-year grade point average (GPA) and persistence to the
 107

second year while a similar small but significant relationship was identified between

participation in educationally purposeful activities and fourth-year GPA. In addition to these

main effects, participation in educationally purposeful activities yielded evidence of conditional

effects within the first-year student cohort with positive effects related to first-year GPA for

students who entered with lower ACT and positive effects related to persistence to the second

year for African American students.

This foundational study gained scholarly prominence via a series of subsequent papers

(Kinzie et al., 2008; Kuh, Kinzie, Buckley, et al., 2007; Kuh et al., 2008) and was also featured

in a 2008 AAC&U monograph entitled High Impact Practices: Who They Are, Who Has Access

to The, and Why They Matter (Kuh, 2008). This monograph also incorporated findings from the

2007 NSSE dataset that specifically examined the relationship between participation in

individual HIP experiences and student learning (National Study of Student Engagement, 2007).

Results of this analysis were mixed with findings suggesting that participation in learning

communities and service learning yielded the most substantive gains in deep learning for the

first-year cohort while an effect of similar magnitude was identified for participation in

undergraduate research for the senior year cohort. The relationships between participation in

other HIPs and deep learning were generally positive and more moderate in magnitude.

With early findings signaling benefits of HIP participation, Kuh (2008) recommended

that undergraduate students participate in at least two HIP experiences with one situated early in

their enrollment and one sequenced closer to graduation. Using NSSE data collected from 2006-

2008, Finley and McNair (2013) identified a positive relationship between increased

participation in HIP experiences and self-reported gains in general education, practical

 108

competence, and personal and social development. Although these patterns were reflected both

in aggregate with the full analytic sample and when considering differences across student

race/ethnicity, gains appeared to be particularly salient for first-generation students with self-

reported gains in each of the aforementioned areas exceeding those of continuing generation

students (Finley & McNair, 2013).

Extending the HIP Discourse

Despite the positive effects of participation in HIP experiences identified in early studies,

the literature was not without limitation related to sample selection, methodology, and outcomes

(Brownell & Swaner, 2010). Moreover, as researchers sought to address some of these

limitations, the relationship between participation in HIP experiences and college outcomes

became more nuanced. More recent studies that use the NSSE dataset offer continue to provide

evidence that learning gains vary by HIP experience (Valentine et al., 2021) Additionally, these

gains vary not just by type but also magnitude based on student race/ethnicity (Valentine et al.,

2021; Zilvinskis, 2019). A similar finding emerged related to post-graduation plans with

differential effects identified across HIP experiences (Miller et al., 2018).

As the literature base expanded to include studies with multi-institutional datasets other

than the NSSE, further evidence that clarified and sometimes contradicted the prevailing

discourse emerged. Both the Wabash National Study of Liberal Arts Education (WNS) and the

Educational Longitudinal Study (ELS) offered researchers variables from multiple waves of data

collection, thereby permitting a broader range of pre-college characteristics to be incorporated

into analytic models and more diverse sets of college outcomes to be considered. Studies using

the WNS contributed to the growing body of evidence that learning gains vary across individual
 109

HIP experiences (Kilgo et al., 2015; Seifert et al., 2014) and that an overemphasis on general

effects may mask interaction effects for minoritized student populations (Seifert et al., 2014).

Little consensus emerged regarding the relationship between HIP participation and college

completion with some evidence suggesting no relationship exists (Johnson & Stage, 2018) and

other findings suggesting not only a positive effect but also a cumulative one (McDaniel &

VanJura, 2020). Finally, post-graduation outcomes began to be explored with positive and

cumulative effects of HIP participation identified for civic engagement in early adulthood

(Myers et al., 2018).

In the only study to explore the relationship between HIP participation and early career

outcomes, Wolniak and Engberg (2019) used the ELS dataset to study the relationship between

participation in internships, research with a faculty member, study abroad, community-based

projects, and capstone experiences and early career outcomes such as self-reported earnings and

attitudes toward their current jobs. The OLS regression models that informed this analysis

offered no evidence of consistent general effects of participation in HIP experiences. Rather, the

relationship between participation across the set of HIP experiences and early career outcomes

was found to vary and was not uniformly positive. However, interaction effects based on college

major were identified with participation in different HIP experiences yielding different post-

graduation outcomes for different majors. Moreover, both self-reported earnings and early career

attitudinal outcomes are positively influenced by the congruence between student field of study

and professional employment (Wolniak & Engberg, 2019).

Despite the expansion of the literature base relating to HIP participation and college

outcomes, few studies have specifically accounted for the issue of student choice (Astin, 1984), a
 110

notable omission as participation in HIP experiences is not universally required for graduation.

Observational studies that do not account for the opt-in nature of these activities are subject to

issues of selection bias (Rosenbaum, 2002) in that students who elect to participate in a particular

experience may be different from those who choose not to participate. Propensity score (PS)

analysis offers researchers a variety of techniques including matching, weighting, and

stratification to reduce bias (Austin, 2011; Stuart, 2010; Thoemmes & Kim, 2011). While many

studies include covariates in their respective analyses, PS techniques are more advantageous in

that they permit group differences to be balanced independent of the outcome under study,

thereby reducing the potential for error (Steiner et al., 2010). Indeed, the studies that have

incorporated PS adjustments in the examination of the relationship between HIP participation

and college outcomes (Bowman & Holmes, 2018; Culver & Bowman, 2019; Provenchar &

Kessel, 2019) have generally found that the absence of such adjustments resulted in

overestimation of the effects of HIP participation.

Limitations in Previous Research

While considerable progress has been made to address some of the limitations in the

foundational HIP scholarship (Brownell & Swaner, 2010), further inquiry is warranted. The

dominance of observational studies that do not account for selection bias via PS analytic

techniques or other methods in both foundational and contemporary scholarship is particularly

notable (Finley & McNair, 2013; Kilgo et al., 2014; Kuh, Kinzie, Cruce, et al., 2007; Kuh, 2008;

McDaniel & Van Jura, 2020; Miller et al., 2018; Myers, et al., 2018; Seifert et al., 2014;

Valentine et al., 2021; Wolniak & Engberg, 2019). Moreover, the studies that incorporated these

types of controls likewise exhibited some of the limitations identified by Brownell and Swaner
 111

(2010) in that they used data from a single institution (Provenchar & Kassel, 2019) or focused on

a single HIP experience (Bowman & Holmes, 2018; Culver & Holmes, 2019). Additionally, few

studies have identified benefits of cumulative participation in HIP experiences (Finley &

McNair, 2013; McDaniel & Van Jura, 2020; Myers et al., 2018) and of those, just one has

focused on post-graduation outcomes (Myers et al., 2018).

Current Study

The present study was designed to extend existing scholarship on the relationship

between participation in two or more HIP learning experiences as recommended by Kuh (2008)

and post-graduation career attitudes using the dataset from the Educational Longitudinal Study of

2002. With a focus on the relationship between cumulative participation in HIP experiences and

perceptions of learning and challenge in professional experiences, the present study was guided

by the following research questions:

1. Does participation in two or more HIP experiences affect perceptions of learning

and challenge in professional experiences when controlling for student precollege

sociodemographic characteristics, high school academic preparation,

organizational factors, and curricular and cocurricular individual student

experiences?

2. Is the relationship between participation in two or more HIP experiences and

perceptions of learning and challenge in professional experiences moderated by

student race/ethnicity, student sex, student SES, and high school academic

achievement?
 112

The present study explored these research questions via PS analysis using structural

equation modeling (SEM), a methodological approach developed and tested by Leite et al.

(2019). Using MPlus software, this approach incorporated the ability to account for self-selection

using PS analysis as well as the complex relationships among diverse variables as represented by

contemporary college outcomes scholarship (Reason, 2009) using SEM. With a multiple-group

SEM design, the methodology allowed for examination of differences between students who

participated in two or more HIPs and those who did not. Finally, the use of sensitivity analysis

permitted consideration of the degree to which omitted variables may influence the results of

model testing (Harring et al., 2017).

Data Source

Sponsored by the National Center for Educational Statistics (NCES), the Educational

Longitudinal Study (ELS) served as the data source for the present study. It was designed to

examine moments of significant transition in students’ educational experiences beginning in high

school and continuing into postsecondary education as well as their careers. High school

completion, college access and persistence, and postsecondary outcomes were central foci of the

ELS dataset along with college access and choice. The ELS aggregated information related to

factors that might influence decision making at these points of transition including student

background, family support, high school experiences, postsecondary experiences, and

employment experiences as well as high school, college, and post-college outcomes (Ingels et

al., 2004).

The ELS dataset was particularly suited to the present study as it contained variables that

correspond with a range of factors associated with both traditional (Astin, 1984; Astin, 1993;
 113

Tinto, 1993; Terenzini et al., 1997) and contemporary college outcomes scholarship (Reason,

2009; Ro et al., 2013; Terenzini & Reason, 2005). Information pertaining to students’ precollege

background including sociodemographic characteristics, academic preparation and performance

in high school, and student dispositions was present in the dataset. Structural-demographic

organizational factors were likewise present as were curricular and cocurricular individual

student experiences. That said, other elements of particular focus in the contemporary college

outcomes discourse (Reason, 2009; Ro et al., 2013) including organizational behavior measures,

peer environment indicators such as campus racial climate and campus academic climate, and

individual classroom experiences were not included and remain topics for future research.

Of particular relevance were the ELS items related to HIP participation. A composite

variable representing cumulative student participation in a range of HIP experience types as well

as dichotomous indicators representing individual participation in internships, research, study

abroad, community-based projects, and capstone experiences were included in the ELS dataset.14

While these were indirect measures that relied solely on self-reported participation, they served

as indicators of participation in certain types of HIPs and facilitated understanding of cumulative

participation across different types of experiences.

The ELS data source contained variables collected from diverse sources over four rounds

of data collection across 15 years (Ingels et al., 2004). Base year data collection began in 2002

which represented the sophomore year of high school for the cohort. Subsequent data collection

14
The HIP participation measures in the ELS are distinct among available NCES datasets. Collected in 2012, this
data was released for public use in 2014. While more recent studies including the Beginning Postsecondary Students
Longitudinal Study (BPSL) of 2012 and the HSLS: 2009 included measures of HIP participation, neither was as
comprehensive across diverse experience types as the ELS. The BPSL dataset included questions about participation
in study abroad but not internships, research, community-based projects, or capstone experiences. The HSLS:2009
dataset included only three HIP activities: research, study abroad, and community-based projects.
 114

consisted of the first follow-up questionnaire in spring 2004 (Ingels et al., 2005), the second

follow-up questionnaire in spring 2006 (Ingels et al., 2007), and the third and final follow-up

questionnaire in summer 2012 (Ingels et al., 2014). High school transcripts were collected during

the 2004 first follow-up and postsecondary transcripts were collected during the 2012 third

follow-up.

Variables for the present study were derived from data collected during the second, third,

and fourth waves of data collection. Variables were extracted from the 2004 first follow-up

questionnaire, high school transcript, the 2006 second follow-up questionnaire, and the 2012

third follow-up questionnaire. The largest percentage of variables was selected from the third

follow-up questionnaire given the focus on postsecondary experiences and post-graduation

college outcomes as shown in Appendix B.

Analytic Sample

The analytic sample for the present study consisted of the ELS twelfth grade student

cohort which was representative of the population of high school seniors in 2004 (Ingels et al.,

2014).15 Moreover, with research questions focused on post-graduation college outcomes, only

students who had earned a bachelor’s degree or higher at the time of the third follow-up

questionnaire were included in the analytic sample (F3ATTAINMENT  6 (bachelor’s degree)).

This approach was consistent with existing scholarship that has previously used the ELS dataset

to examine the relationship between HIP experiences and early career outcomes (Wolniak &

Engberg, 2019). These specifications yielded a prospective analytic sample in which 38.66% of

15
Since a third follow-up weight was employed in the present study per NCES guidelines, the G12 cohort should
include respondents who were designated as cohort members during the first follow-up (G12COHRT =1) as well as
those who were not designated as cohort members until the second follow-up (G12COHRT =2) (Ingels et al., 2014).
 115

cases had data missing on at least one variable with 166 missing data patterns. Given both the

nature of the data structure and the complexity of the analytic techniques, full case analysis was

employed for the present study, yielding a final analytic sample of 3,037 cases.

Missing data is a common issue in survey datasets related to postsecondary experiences

(Cheema, 2014) and as such, researchers frequently deploy a variety of techniques to address

these challenges including expectation-maximization (EM), full-information maximum

likelihood (FIML) and multiple imputation (MI). Although EM, FIML, and MI were evaluated

for prospective use for the present study consistent with relevant scholarship (Cox et al., 2014),

the complexity of the analytic techniques raised further challenges. While multiple studies have

addressed the issue of combining PS estimates generated by MI (Hill, 2004; Mitra & Reiter,

2016), there is little empirical support for procedures to combine fit indices generated by MI

(Enders & Mansolf, 2016). Additionally, the categorical nature of multiple covariates in the

model precluded the use of EM and FIML at distinct points in this analysis, ultimately resulting

in the use of full case analysis. While not the most sophisticated approach currently available to

researchers, full case analysis can be reasonably employed under the right conditions (Cox et al.,

2014) related to the nature of the missingness (Allison, 2002) and sufficiency of included

covariates (Graham, 2009).

Within the analytic sample, 69.25% identified as White while 11.95% identified as Asian,

6.65% as Black or African American, 7.97% as Hispanic, 3.82% as multiracial, and 0.36% as

American Indian/Alaska Native. 58.25% identified as female with 41.75% identifying as male.

Mean SES was 0.48 with a standard deviation of .71 and a range from -2.12 to 1.97. 76.97%

earned a cumulative high school GPA of 3.0 or above with 42.54% earning a high school GPA
 116

of 3.5 or above. 35.96% of analytic sample members planned to graduate with a bachelor’s

degree while an additional 37.21% expected to attain a master’s degree or equivalent, and

22.92% expect to earn a PhD, MD, or other advanced degree as of their senior year of high

school. Many analytic sample members (68.62%) attended a public institution as their first

postsecondary institution. A larger majority (87.22%) enrolled in a four-year institution as their

first postsecondary institution. Most completed their studies at their first postsecondary

institutions as 34.97% of respondents in the analytic sample transferred from their first

postsecondary institutions.

Outcome Measure

The present study employed a cross-sectional design with a post-graduation latent16

outcome measure focused on perceived learning and challenge in professional experiences

(LEARN). This variable was derived from ELS items: F3B25B, F3B25D, and F3B25F with the

item stem “Please indicate to what extent the following job characteristics [apply/applied] to

your [current/most recent job].” (National Center for Educational Statistics, n.d.). Collected via

the 2012 third follow-up questionnaire, F3B25B corresponded with opportunities to learn new

things, F3B25D indicated new challenges, and F3B25F denoted societal impact (National Center

for Educational Statistics, n.d.). These items were measured on a five-point scale that ranged

from definitely not an aspect of the respondent’s current/most recent job (1) to very much an

aspect of the respondent’s current/most recent job (5). For the analytic sample, Coefficient Alpha

for the LEARN latent outcome measure was 0.80.

16
Latent variables are measured indirectly using responses from a set of questions or test items (Schumacker &
Lomax, 2016), differing from observed variables which can be measured directly. The use of both observed and
latent variables in SEM models permits researchers to account for substantive complexity in relationships between
variables (Schumacker & Lomax, 2016; Streiner, 2006; Wang & Wang, 2012).
 117

Treatment Measure

The treatment measure was a dichotomous variable that represented student participation

in two or more types of HIP experiences while enrolled as an undergraduate student at a

postsecondary institution (HIPREC). This variable was calculated using ELS items F3A14A-

F3A14E with the item stem “Have you participated/Did you participate] in any of the following

as a part of your [undergraduate/college] education?” (National Center for Educational Statistics,

n.d.). HIP experience data collected during the 2012 third follow-up included (a) internship, co-

op, field experience, student teaching, or clinical assignment (F3A14A), (b) research project with

a faculty member outside of course or program requirements (F3A14B), (c) study abroad

(F3A14C), (d) community-based project as part of a regular course (F3A14D), and (e)

culminating senior experience, such as a capstone course, senior project or thesis, or

comprehensive exam (F3A14E) (National Center for Educational Statistics, n.d.). ELS variables

F3A14A-F3A15E were dichotomous indicators with response options as follows: no (0) and yes

(1). These variables were included in the third follow-up questionnaire administered in 2012.

For the treatment measure (HIPREC) in the present study, sample members were coded

as 0 if they responded yes to fewer than two of the ELS F3A14A-F3A15E items. They were

coded as a 1 if they responded yes to two or more of the ELS F3A14A-F3A15E questions.

Within the analytic sample, the group of respondents who reported two or more HIP experiences

during their undergraduate career were considered to be the treatment group (1) as they

completed at least the recommended number of HIP experiences (Kuh, 2008). Correspondingly,

the group of respondents who indicated that they had less than two HIP experiences during their

undergraduate career were designated as the control group (0). 54.59% of analytic sample
 118

members reported participation in two or more HIP experiences during their undergraduate

enrollment.

Model Covariates

The 24 covariates in this model represented a variety of domains that were designed to

promote the efficacy of PS analysis in accounting for selection bias (Steiner et al., 2015) and

support the use of full case analysis (Graham, 2009). These domains included student precollege

characteristics, organizational factors, and individual student experiences that have been

identified as influential to student learning and persistence in contemporary theoretical

frameworks related to college outcomes (Reason, 2009; Terenzini & Reason, 2005). Please see

Appendix B for a list of all covariates including a brief description of each as well as the ELS

reference variables that informed variable specification.

As outlined by Reason (2009), student sociodemographic traits such as race/ethnicity,

student sex, and student SES are important element within the domain of precollege

characteristics. In this model, race/ethnicity (SCSD-RACE) was represented by a dichotomous

covariate in which American Indian/Alaska Native, non-Hispanic; Black or African American,

non-Hispanic; Hispanic, no race specified; Hispanic, race specified; and More than one race,

non-Hispanic were recoded as members of minoritized populations (1). Asian, Hawaii/Pacific

Islander, non-Hispanic and White, non-Hispanic were coded as members of non-minoritized

populations (0). These groupings reflect evidence of higher persistence for Asian and White

students than members of other populations (Reason, 2003). Biological sex (SCSD-SEX) was

likewise a dichotomous variable with female (0) and male (1) categories. Socioeconomic status

(SCSD-SES) was a composite continuous measure which included indicators of total family
 119

income from all sources in the base year and the highest level of education attained by the

student’s mother and father along with measures of occupational prestige for both the mother and

father as of the 2004 first-follow-up (National Center for Educational Statistics, n.d.).

A less explored aspect of student precollegiate characteristics and experiences pertains to

support from family and friends for college enrollment, an area that Reason (2009) particularly

noted as warranting further exploration. The degree to which students perceive support for

postsecondary enrollment after high school was included as a latent covariate (SCSD-FAMSUP)

representing student perceptions of support from their mother, father, close relative, and friend

Individuals who selected a don’t know response option were recoded as 0 to reflect an absence of

knowledge about the aspirations of the respondent’s family and friends (Lombardi et al., 2011).

The response option noting that respondents should do what they want to do was recoded as a 1.

Support for all other non-college options including a full-time job, vocation or apprenticeship,

military service, and marriage were collapsed into a single non-college category (2). The college

enrollment response option was recoded as college enrollment (3). For the analytic sample,

Coefficient Alpha for this latent covariate was 0.70.

With indicators of academic preparation and performance among the strongest predictors

of college outcomes (ACT, 2007), the present study incorporated both an observed covariate

representing academic performance and a latent covariate that reflected students’ perception of

the degree to which their high school experiences prepared them for postsecondary education.

With evidence signaling a positive and consistent relationship between high school GPA and

college graduation (Adelman, 2006; Allensworth and Clark, 2020), the SCHS-GPA measure

served as an indicator of student’s prior academic achievement, capturing cumulative GPA from
 120

ninth grade to twelfth grade. The SCHS-PREP latent covariate measured respondent perceptions

of the degree to which their high school classes in math, science, and English/writing prepared

them for enrollment at their first postsecondary institution. For the analytic sample, Coefficient

Alpha for this latent covariate was 0.51.

Given prior scholarship documenting the positive relationship between academic goals

and college persistence (Brown et al., 2008; Robbins et al., 2004), the SCD-GOAL measure

served as a dispositional indicator of student’s educational goals as of their senior year in high

school in 2004. Respondents indicated the highest level of education that they expected to

achieve: less than high school graduation (1), GED or other equivalency only (2), high school

graduation only (3), attend or complete 2-year college/school (4), attend college, 4-year degree

incomplete (5), graduate from college (6), obtain master’s degree or equivalent (7), obtain PhD,

MD, or other advanced degree (8), and don't know (9). For purposes of this analysis, the don’t

know (9) response option was recoded as a 0, reflecting an absence of certainty related to

educational goals at the time of measure (Lombardi et al., 2011).

Organizational context covariates were derived from the structural-demographic

characteristics of the first postsecondary institution attended by the student respondent. The

inclusion of these covariates reflected the significant influence of the first institution on

educational attainment (Pascarella & Terenzini, 2005). Institutional control (ORG1-CON) was a

dichotomous indicator representing public (0) and private (1) institutions. Similarly, institutional

level (ORG1-LEV) was a dichotomous indicator representing two-year institutions (0) and four-

year institutions (1).

 121

The individual student experience domain for the present study necessarily included

curricular covariates that reflected both student enrollment and field of study. Consistent with

recent scholarship that identified interaction effects of HIP participation on post-graduation

college outcomes that varied by student major (Wolniak & Engberg, 2019), six dichotomous

covariates served as indicators of the academic field of study for the student’s first known

bachelor’s degree (see Appendix C). Additionally, transfer status relative to first postsecondary

institution (EXPCU-TSFR) was included as a dichotomous covariate indicating transfer status:

no credential earned from and no longer attending first postsecondary institution (1) and

credential earned or still attending first postsecondary institution (0).

The final domain of individual student experiences included covariates that reflected

experiences that occurred outside the classroom. Cocurricular experiences during the first two

years of enrollment included faculty interaction outside of class meetings (EXPCO-FAC),

academic advisor engagement (EXPCO-ADV), participation in intramural or nonvarsity sports

(EXPCO-INT), participation in varsity or intercollegiate sports (EXPCO-VAR), and

participation in extracurricular activities (EXPCO-EXT). Employment was captured as two

separate continuous covariates noting the number of jobs held during the first (EXPCO-JOB1)

and second (EXPCO-JOB2) years of enrollment. Since lack of a mentor has been identified as a

barrier to HIP participation (Finley & McNair, 2013), a variable representing access to a mentor

(EXPCO-MEN) was also included as a dichotomous indicator indicating access (1) or lack

thereof (0).

Weights
 122

In order to account for the complexities of the longitudinal design including unequal

patterns of selection and nonparticipation from selected sample members (Ingels et al., 2014), the

F3FITSCWT sampling weight was applied in alignment with NCES guidelines for analyses with

the G12COHORT that includes variables from the first follow-up questionnaire, the high school

transcript data collection, the second follow-up questionnaire and the third follow-up

questionnaire (Ingels et al., 2014). Additionally, balanced repeated replication (BRR) weight

adjustments were applied to facilitate accurate estimations of standard errors (McNeish et al.,

2017; Stapleton, 2008). In order to support these adjustments, six sets of 200 BRR weights were

computed by NCES consistent with the agency’s guidelines (Ingels et al., 2014). This guidance

further specified that F3F1T001-F3F1T200 BRR set corresponds with the selected F3FITSCWT

sampling weight.

Methods

The present study employed propensity score analysis (PS) analysis with structural

equation modeling (SEM) using MPlus software (Muthén & Muthén, 2017) following the

analytic approach proposed by Leite et al. (2019). Due to the categorical nature of many of the

covariates in the model, models were fit with weighted least squares with adjusted means and

variances (WLSMV) estimation (Beauducel & Herzberg, 2006). They were run with and without

replication weights. While the models were evaluated for exact fit using a modified Chi-Square

(χ2) (Stapleton, 2008), conventional practice in SEM stipulates the use of multiple indices to

assess fit (Schumacker & Lomax, 2016). Fit indices recommended by Leite et al. (2019) for

these purposes include the Comparative Fit Index (CFI), the Tucker-Lewis Index (TLI), and the
 123

Root-Mean-Square Error of Approximation (RMSEA). For all steps of this analysis, fit was

determined to be close if CFI  0.95, TLI  0.95, and RMSEA  0.05.

Step 1: Multiple-Group Confirmatory Factor Analysis (CFA) for Latent Covariates

Following the data preparation, a multiple-group CFA was fit in MPlus (Muthén &

Muthén, 2017) with the two latent covariates (SCSD-FAMSUP and SCHS-PREP) and the

treatment and control groups to verify the factor structure of the latent covariates. Additionally,

measurement equivalence of the latent covariates across treatment and control groups was

evaluated. Known as scalar invariance (Millsap & Olivera-Aguilar, 2012), this condition allows

researchers to appropriately compare means and variances across the treatment and control

groups and is a required for mitigating selection bias in PS analysis (Leite et al., 2019). In

MPlus, scalar invariance is tested with an adjusted likelihood ratio test (Asparouhov et al., 2006).

In this case, Chi-Square (χ2) served as the relevant test statistic; an absence of statistical

significance indicates that the assumption of scalar invariance has been met.

Step 2: SEM for PS Estimation

Once the factor structure and scalar invariance were confirmed for the latent variables,

the SEM depicted in Appendix D was fit in MPlus (Muthén & Muthén, 2017) to calculate PS

weights. This model was hypothesized to represent variables that have been demonstrated to

correlate with participation in HIP experiences in the college outcomes literature. The inclusion

of these covariates was designed to mitigate the issue of selection bias associated with the opt-in

nature of HIP experiences (Steiner et al., 2010).

The SEM was run with the latent and observed covariates as predictors and the HIPREC

treatment measure as the outcome. Given that the literature suggests a cumulative benefit of HIP
 124

participation to participants in increasing numbers of HIP experiences (Finley & McNair, 2013;

McDaniel & Van Jura, 2020; Myers et al., 2018), PS weights were estimated using average

treatment effect on the treated (ATT) weights (Heckman, 2005). The PS weights for the

treatment and control groups were then compared to evaluate the assumption of overlap in PS

analysis. For this analysis, the assumption of overlap was determined to have been met if neither

group had a PS with a value of 0 or 1 (Leite et al., 2019).

Step 3: Latent Covariate Balance Evaluation and Step 4: Observed Covariate Balance

Evaluation

Following the calculation of the PS weights, these values were multiplied by the

F3FITSCWT sample weight to obtain final PS weights. Covariate balance was evaluated using

regression models of the latent and observed covariates on the treatment measure. This

assessment permitted the researcher to compare the distributions of the treatment and control

groups following application of the final PS weights via examination of the standardized mean

differences across the regression coefficients (Leite et al., 2019). This assessment was performed

using two sets of criteria given a lack of consensus in the literature. Austin (2011) stipulates

absolute values of standard mean differences below 0.100 standard deviations as evidence of

adequate covariate balance while Stuart and Rubin (2007) designate .250 as an appropriate

threshold by which to evaluate covariate balance.

Step 5: Multiple-Group CFA for Latent Outcome Using PS Weights

Following the estimation of the PS weights, a multiple-group CFA was fit in MPlus

(Muthén & Muthén, 2017) for the outcome measures using the final PS weights with the sample

weight adjustments. These were run with the adjusted LEARN latent outcome variables and the
 125

treatment and control groups. This CFA was used to affirm the LEARN factor structure and

evaluate scalar invariance (Millsap & Olivera-Aguilar, 2012) of the outcome measure across the

treatment and control groups using the adjusted likelihood ratio test (Muthén & Muthén, 2017).

As with the previous scalar invariance analysis, Chi-Square (χ2) served as the relevant test

statistic with an absence of statistical significance indicating that the assumption of scalar

invariance was met.

Step 6: Multiple-Group CFA for ATT Estimation Using PS Weights

In addition to testing for scalar invariance, the multiple-group CFA was used to estimate

the ATT using PS weights. As noted in Step 5, this model was run with the adjusted LEARN

variable and the treatment and control groups. The mean of the outcome measure for respondents

with no HIP experiences was set to zero to identify the model. With this adjustment, the resultant

mean of the outcome measures for respondents who have participated in two or more HIP

experience represented the latent mean difference between the groups, i.e., the ATT (Leite et al.,

2019).

Step 7: Multiple-Group SEM for Interaction Effect Estimation

A multiple-group SEM was used to evaluate interactions (Wang & Wang, 2012) between

the outcome measure and the following observed covariates: race/ethnicity (SCSD-RACE), sex

(SCSD-SEX), SES (SCSD-SES), and high school GPA (SCHS-GPA). In this model,

participation in two or more HIP experiences (HIPREC) served as a grouping variable with

sample members who completed two or more HIPs considered to be the treatment group and

those who completed fewer than two HIPs designated as the control group (see Appendix E).

The estimated differences for the coefficients between the groups were used to determine the
 126

nature of any identified interactions. Although no covariates were needed due to the presence of

the final PS weights, linear, cubic, and quadratic PS estimates were added to achieve doubly

robust estimation and account for any lingering bias in the model (Bang & Robins, 2005; Schafer

& Kang, 2008).

Step 8: Multiple-Group SEM with Phantom Variable for Sensitivity Analysis

Using the PS weights, a multiple-group SEM with a phantom variable was fit in MPlus

(Muthén & Muthén, 2017) to evaluate in the sensitivity of the model to omitted confounders

using the methodology specified by Harring et al. (2017). Consistent with Leite et al. (2019),

nine multiple-group SEMs were used with varying permutations of sensitivity parameters. The

first parameter which corresponded with SMD on the phantom variable was tested with

parameters of 0.25, 0.5, and 1.0. It was tested in combination with the second parameter which

consisted of the standardized path from the phantom variable to the outcome measure, likewise

with parameters of 0.25, 0.5, and 1.0. If the p values for any of the interactions across the

multiple-group SEMs demonstrated a change in statistical significance (i.e., from non-

significance to statistical significance) following the parameter adjustment, it was interpreted to

signal the absence of potential confounders.

Results

The multiple-group CFA provided evidence to support the factor structure of latent

covariates SCSD-FAMSUP and SCHS-PREP. As shown in Table 5, the fit indices signal close,

albeit not exact, fit under conditions of both configural and scalar invariance. The adjusted

likelihood ratio test (Asparouhov et al., 2006) that served as a comparison between configural

and scalar invariance was not statistically significant (χ2 (14) = 15.059, p = .374). The lack of
 127

statistical significance indicated that the assumption of scalar invariance had been met, thereby

permitting comparison of means and variances across the treatment and control groups.

Table 5

Manuscript II: Model Fit Statistics and Indices

Estimation of the PS scores with the SEM presented in Appendix D likewise yielded

model fit that was close but not exact as shown in Table 5. Analysis of the descriptive statistics

for the PS scores further indicated that the requisite assumption of overlap was met with neither

group containing a PS score of zero or one. For students who participated in two or more HIPs,

minimum, maximum, and mean propensity scores were .119, .966, and .640. Respective analogs

for students who did not participate in two or more HIPs were .124, .938, and .480.

Following PS estimation, PS weights for the ATT were calculated by multiplying the PS

estimates by the F3FITSCWT sample weight. Inspection of the PS weights for the ATT yielded a

maximum value of 15.129. Since this value did not exceed the sample size, it was assumed that

extreme weights were unlikely to impact subsequent analyses (Leite et al., 2019). As shown in

Table 6, the results of the covariate balance evaluation for both the latent and observed

covariates mostly demonstrated adequate ATT-weighted balance. Absolute values for the

standardized means differences were equal to less than .100 for all but three covariates (i.e.,

EXPCU-EDU, EXPCU-HHS, EXPCU-SS) (Austin, 2011) but were less than .250 for all model

covariates (Stuart & Rubin, 2007).

 128

Table 6

Manuscript II: Standardized Mean Differences (SMD) for Covariate Balance Evaluation

Covariate ELS Source Description Baseline ATT

Variable a SMD weighted
SMD
Student Precollege Characteristics and Experiences
Sociodemographic traits
SCSD-RACE F1RACE Membership in a minoritized racial population -.102 -.007
SCSD-SEX F1SEX Biological sex -.155 -.018
SCSD-SES F1SES2 SES .179 .006
SCSD-FAMSUP F1S44A-F1S44D Perception of support network aspirations .038 .004

Academic performance and preparation

SCHS-GPA F1RGPP2 High school GPA .245 -.003
SCHS-PREP F2B17A-F2B1C Perception of academic preparedness .200 -.024

Student dispositions
SCD-GOAL F1STEXP Postsecondary educational expectations .154 .004

Organizational Context
ORG1-CON F3PS1SEC Control of first postsecondary institution .467 -.009
ORG1-LEV F3PS1SEC Level of first postsecondary institution .365 .044

Individual Student Experiences

Curricular experiences
EXPCU-TSFR F3PS1RETAIN Student respondent’s transfer status -.189 -.059

Field of study for first known bachelor’s degree

EXPCU-ART F3TZBCH1CIP2 Arts and humanities .034 .002
EXPCU-BUS F3TZBCH1CIP2 Business -.476 -.008
EXPCU-EDU F3TZBCH1CIP2 Education .080 .115
EXPCU-HHS F3TZBCH1CIP2 Health and human services .229 -.108
EXPCU-SS F3TZBCH1CIP2 Social science .133 .101
EXPCU-STEM F3TZBCH1CIP2 Science, technology, engineering, and math .110 -.067

Out-of-class experiences
EXPCO-FAC F2B18A Talk with faculty outside of class .321 .009
EXPCO-ADV F2B18B Meet with their advisor about academic plans .145 .058
EXPCO-INT F2B18E Participate in intramural or nonvarsity sports .132 .019
EXPCO-VAR F2B18F Participate in varsity or intercollegiate sports -.031 -.073
EXPCO-EXT F2B18G Participate in extracurricular activities .438 -.063
EXPCO-JOB1 F2C24_P Number of jobs while enrolled in 2004-2005 -.025 -.023
EXPCO-JOB2 F2C29_P Number of jobs while enrolled in 2005-2006 .033 .016

EXPCO-MEN F3A14F Access to mentoring experience .801 -.040

a
The first two characters of the ELS source signal the corresponding data collection wave: F1 variables were
collected in 2004, F2 variables in 2006, and F3 variables in 2012.
 129

Once the adequacy of covariate balance was established, a multiple-group CFA was

conducted to verify the factor structure of the LEARN latent outcome variable. As shown in

Table 5, the fit indices provide evidence of close but not exact fit under conditions of both

configural and scalar invariance. With a non-significant result for the adjusted likelihood ratio

test comparing configural and scalar invariance (χ2 (10) = 11.422, p = .326), the assumption of

scalar invariance of the adjusted LEARN latent outcome variable was determined to have been

met (Asparouhov et al., 2006).

The multiple-group CFA with scalar invariance was run with the adjusted latent outcome

variable and the treatment and control groups to estimate the ATT using PS weights. With the

mean of the outcome measure for respondents without the recommended number of HIP

experiences set to zero, the mean of the outcome measures for respondents who participated in

two or more HIP experiences represented the ATT estimate (Leite et al., 2019). The ATT

estimate of the perceived affinity for learning and challenge in professional experiences

(LEARN) was not statistically significantly different from zero without BRR weights ( = .107,

SE = .166, p = .519). When BRR weights were used to improve the accuracy of the standard

error (SE) estimation, the corresponding ATT estimate remained not statistically significant ( =

.107, SE = .071, p = .133).

Following the estimation of the multiple-group CFA, a multiple-group SEM to estimate

the interaction effects under study was run. Linear, quadratic, and cubic PS estimates were added

to the model for a doubly robust ATT estimation. As shown in Table 5, this model demonstrated

close but not exact fit. The doubly robust ATT estimate of the perceived learning and challenge

in professional experiences (LEARN) was not statistically significantly different from zero
 130

without BRR weights ( = -.121, SE = 1.068, p = .909). Likewise, when BRR weights were

added, the corresponding ATT estimate was not statistically significant ( = -.121, SE = .263, p =

.644).

Just as the main effect was not statistically significant, no interactions were found to be

statistically significantly different from zero as shown in Table 7. For purposes of this analysis, a

group difference that was statistically significant would have served as evidence of an

interaction. There was a negative relationship between membership in a minoritized racial group

(SCSD-RACE) and perceived learning and challenge in professional experiences (LEARN) for

students who did not participate in the recommended number of HIPs. Control group respondents

who were members of minoritized racial groups (coded as 1) reported lower perceived learning

and challenge in professional experiences (LEARN) than respondents who were not members of

minoritized racial groups (coded as 0). Similarly, there was a nearly equivalent negative

relationship between membership in a minoritized racial group and LEARN for students who

participated in two or more HIPs. The difference between these coefficients was not statistically

significant (p = .788).

Table 7

Manuscript II: Interaction Estimates from Multiple-Group SEM

Coefficients with ATT weights

Group No HIPREC (SE) HIPREC (SE) Difference (SE)
SCSD-RACE -.087* (.037) -.082* (.039) -.019 (.069)
SCSD-SEX .087* (.041) .048 (.034) .056 (.066)
SCSD-SES .022* (.026) -.011 (.019) .039 (.039)
SCHS-GPA .041 (.020) -.001 (.018) .051 (.031)
Propensity score .363* (.137) .706* (.244)
Propensity score squared -.528* (.214) -1.124* (.535)
Propensity score cubed .843* (.258) .556 (.293)
* Denotes statistically significant coefficients at  = .05
 131

Similarly, as shown in Table 7, there was no evidence of statistically significant

interaction effect related to biological sex (SCSD-SEX) (p = .401). There was a positive

relationship between biological sex and perceived learning and challenge in professional

experiences (LEARN) for students who did not participate in the recommended number of HIPs.

In this analysis, male respondents (coded as 1) who participated in fewer than two HIPs reported

higher perceived learning and challenge in professional experiences (LEARN) than female

respondents (coded as 0). For students who participated in two or more HIPs, the relationship

remained positive but decreased in magnitude. Thus, male respondents who participated in two

or more HIP experiences reported lower perceived learning and challenge in professional

experiences than female respondents who likewise participated in two or HIP experiences.

For socioeconomic status (SCSD-SES), there was a mixed relationship between the

socioeconomic status and perceived learning and challenge in professional experiences

(LEARN) as indicated in Table 7. In this model, socioeconomic status was measured on a

continuous scale ranging from a minimum of -2 to a maximum of 2. For students who

participated in fewer than two HIPs, the positive relationship signals that as socioeconomic status

increased, perceived learning and challenge in professional experiences (LEARN) likewise

increased. However, for students who participated in two or more HIPs, the relationship was

negative, meaning that as SES increased, perceived learning and challenge in professional

experiences (LEARN) decreased. However, this difference was not statistically significantly

different from zero (p = .316).

The relationship between high school GPA and perceived learning and challenge in

professional experiences (LEARN) varied across the treatment and control groups as shown in
 132

Table 7. For control group respondents, this relationship was positive; as high school GPA

increased, perceived learning and challenge in professional experiences (LEARN) likewise

increased. However, for treatment group respondents, this relationship was negative albeit small,

meaning that as high school GPA increased, perceived learning and challenge in professional

experiences (LEARN) decreased slightly. However, this difference was not statistically

significantly different from zero (p = .095).

As shown in Table 7, linear, quadratic, and cubic PS estimates were added to the

multiple-group SEM. The inclusion of these scores permits a doubly robust estimation to account

for any remaining bias after the weights were incorporated into the model (Bang & Robins,

2005; Schafer & Kang, 2008). Linear relationships between the PS estimates and perceived

learning and challenge in professional experiences (LEARN) were positive and statistically

significant for the control group (p = .008) and the treatment group (p = .004). Quadratic

relationships were negative and likewise statistically significant for the control group (p = .014)

and the treatment group (p = .033). Cubic relationships for both the treatment and control group

were positive. However, the cubic relationship was statistically significant for the control group

(p = .001) but not statistically significant for the treatment group (p = .057). This signals that the

relationship between probability of participation in the recommended number of HIPs and

perceived learning and challenge in professional experiences (LEARN) may be s-shaped.

Finally, results of the sensitivity analyses are displayed in Table 8. The purpose of this

analysis was to evaluate whether an unspecified omitted covariate would result in change of p-

values for the ATT and interactions across a range of sensitivity parameters. The interactions for

SCSD-RACE and SCSD-SES, which were not statistically significant for the multiple-group
 133

SEM to estimate the interactions, remained not statistically significant across all parameters sets,

thereby signaling that these interaction effects are not influenced by omitted covariates.

However, there was some variation in statistical significance across the parameter sets for the

ATT and the other interaction effects. While the ATT and interaction related to SCSD-SEX were

not statistically significant in the multiple-group SEM to estimate the interaction effects, they

were each statistically significant for one of the parameters as depicted in Table 8, thereby

signaling potential minor sensitivity to omitted covariates for these analyses. Additionally, the

interaction for SCHS-GPA that was not statistically significant in the multiple-group SEM to

estimate the interaction effects was statistically significant across all but one of the parameters.

Thus, the interaction effect for SCHS-GPA demonstrates evidence of high sensitivity to

parameter changes, thereby signaling that it may be influenced by omitted covariates.

Table 8

Manuscript II: P-Values from Sensitivity Analysis

Sensitivity Parameter Pairs

.1, .1 .25, .1 .5, .1 .1, .25 .25, .25 .5, .25 .1, .5 .25, .5 .5, .5
ATT .351 .479 .197 .077 .690 .176 .274 .180 .017*
SCSD-RACE .798 .806 .696 .263 .833 .586 .745 .529 .693
SCSD-SEX .189 .376 .156 .001* .427 .293 .225 .204 .168
SCSD-SES .070 .292 .104 .113 .391 .243 .128 .106 .104
SCHS-GPA .009* .097 .020* .004* .136 .050* .015* .019* .004*
* Denotes statistically significant coefficients at  = .05

Analytical Limitations

Despite its many advantages, several characteristics of the ELS data resulted in analytical

limitations for the present study. While the ELS contained variables that specified participation

in five types of HIP experiences, others were omitted from consideration including first-year

seminars and experiences, common intellectual experiences, learning communities, writing-

intensive courses, and collaborative assignments and projects. Additionally, there were no
 134

variables that specified the number of individual HIP engagements, duration, or context, a

limitation that has been acknowledged in other studies (Myers et al., 2017). Similarly, there were

no indicators pertaining to the quality of those experiences, an area of increasing scholarly

interest within the HIP discourse (Zilvinskis, 2019).

Additionally, the aggregation of the distinct racial/ethnic categories in the SCSD-RACE

variable resulted in limitations to the conclusions that can be drawn in the present study (McNair

et al., 2020). Aggregating these categories particularly limited understanding of the substantive

diversity across student experiences and college outcomes within minoritized student populations

in the U.S. (Patton et al., 2015). However, the racial/ethnic diversity in the analytic sample was

insufficient to permit estimation of the PS scores for each of the individual racial/ethnic

categories.

Finally, several key influences identified within the contemporary college outcomes

scholarship (Reason, 2009) were simply not measured in any of the ELS waves of data

collection. Most notably, while structural-demographic features of postsecondary institutions

were included in the ELS dataset, no variables that measured or approximated the organizational

behavior dimensions of those institutions were incorporated into the ELS dataset. Similarly,

despite the presence of variables that targeted individual student experiences in the curriculum

and co-curriculum, measurement for both was limited to the first two years of enrollment.

Variables related to classroom experiences as well as student peer environments including

campus racial climate and campus academic climate were also absent from the ELS dataset.

Discussion
 135

Despite these limitations, the present study extends the HIP discourse with a

methodology designed to further refine understanding of the impact of cumulative participation

in HIP experiences. The ability to account for the particular issue of selection bias via PS

estimation is important for discerning whether participation in multiple types of HIP experiences

does indeed contribute to a particular outcome or whether such influences are best explained by

differences among individuals who opt to participate versus those who do not, an issue of interest

to both individual students and postsecondary institutions. The doubly robust estimation

employed in this analysis is designed to address any lingering bias that is not otherwise

accounted for by the PS weights (Bang & Robins, 2005; Schafer & Kang, 2008), further

extending the established methodological discourse around PS analysis and HIP participation

(Bowman & Holmes, 2018; Culver & Bowman, 2019; Provenchar & Kessel, 2019) and

providing postsecondary educational researchers with an additional set of rigorous statistical

tools for consideration as they look to evaluate the effects of participation in opt-in learning

experiences such as HIPs.

Moreover, the present study contributes to the scholarship around the range of outcomes

that are examined in the HIP discourse. The foundational scholarship related to the effects of

participation in HIPs was dominated by studies focused on a relatively limited set of outcomes

that were primarily of interest to postsecondary institutions such as persistence, academic

achievement as defined by grade point average (GPA), and graduation rates (Kinzie et al., 2009;

Kuh, 2008; Kuh, Kinzie, Buckley, et al., 2007; Kuh, Kinzie, Cruce, et al., 2007; Kuh et al., 2008)

or on indirect, self-reported measures of student learning (Kuh, 2008; National Study of Student

Engagement, 2007). As more recent scholarship has examined direct measures of student
 136

learning in college as well as postsecondary employment outcomes, the notion of what

constitutes impact has broadened and evidence that suggests that the nature and duration of that

impact may vary has emerged (Myers et al., 2018; Seifert et al., 2014; Kilgo et al., 2015;

Wolniak & Enberg, 2019). Moreover, the use of SEM in the present study to consider latent

outcomes variables such as attitudinal measures of perceived learning and challenge in

professional experiences further expands the range and complexity of outcomes that can be

examined. In an era in which employers perceive benefit to participation in HIPs (Finley, 2021),

there is simultaneously little evidence to suggest uniformly positive effects of HIP experiences

on career-related attitudes (Wolniak & Enberg, 2019). Thus, the ability to explore and examine a

range of attitudinal, cognitive, and behavioral outcomes provides an opportunity to expand the

scope of the value proposition related to participation in HIP experiences beyond measures that

are bounded by a student’s undergraduate enrollment.

Results of this analysis align with and further extend scholarship related to the effects of

participation in HIP experiences on attitudes related to lifelong learning. The relationship

between participation in HIP experiences and general attitudes related to lifelong learning were

previously found to vary across HIP experiences (Kilgo et al., 2015; Seifert et al., 2014). Similar

findings by Wolniak and Engberg (2019) identified varying effects between participation in

individual HIP experiences and perceived learning and challenge in professional experiences.

Just as Wolniak and Engberg (2019) found that participation in any HIP experience did not

correspond with an increase in perceived learning and challenge, the results of this analysis

indicate that even participation in two or more HIP experiences is insufficient to influence

perceived learning and challenge in professional experiences. In this regard, further investigation
 137

is warranted, particularly given prior scholarship that suggests that certain HIP experiences

might be more beneficial for particular majors than others (Wolniak & Engberg, 2019). Thus,

while participation in any two HIPs might be insufficient to enhance perceived learning and

challenge in professional experiences, a particular set of HIPs might have impact for students in

a specific major that would otherwise absent. Thus, the notion of which HIPs, how many, under

what circumstances, and in what order should be examined further to better understand how

these choices might interact in influencing impact for different students.

The results of the present study not only extend the body of evidence related to early

career outcomes (Wolniak & Engberg, 2019) but likewise contribute to scholarship around the

benefits of HIP participation on student learning and development. Studies that have identified a

relationship between self-reported learning gains and HIP participation (Finley & McNair, 2013;

Kilgo et al., 2015; National Study of Student Engagement, 2007; Seifert et al., 2014; Valentine et

al., 2021) used datasets in which measures were taken while students were still enrolled, often

during the first and final years of enrollment. Additionally, studies that found evidence of gain

related to lifelong learning included more types of HIP experiences (Kilgo et al., 2015; Seifert et

al., 2014) beyond the five that were examined in the present study. Conversely, the lack of

statistical significance related to the relationship between perceived learning and challenge in

professional experiences could potentially signal that any perceived benefits to learning and

development may not transfer beyond postsecondary enrollment. This finding is consistent with

the learning transfer literature noting the particular challenge associated with transferring

attitudes and skills from one context to another due to the specific nature of many learning

experiences (Ambrose et al., 2010). These findings may carry particular relevance for educators
 138

who develop, administer, and assess programs such as general education curricula that frequently

target attitudes, skills, and behaviors related to the lifelong learning.

Additionally, the results of the sensitivity analysis point to the need for further research

that incorporates additional covariates that may be absent from the model used in the present

study. Current scholarship provides insight into prospective domains of interest for additional

inquiry. Variables related to experiential features, implementation fidelity, and overall quality

within HIP experiences may be particularly germane (Kuh & Kinzie, 2018; Zilvinskis, 2019).

While not present in the ELS dataset, domains of interest in contemporary college outcomes

scholarship including organizational behavior dimensions of postsecondary institutions, students’

classroom experiences, and peer environments (Reason, 2009) may also be worthy of further

exploration. Finally, variables that offer additional data points related to post-college experiences

and career outcomes could be incorporated into the model including career-related measures

such as salary and job satisfaction (Wolniak & Engberg, 2019) and additional sociodemographic

variables such as marital status, number of dependents, and geographic region (Myers et al.,

2018; Wolniak & Engberg, 2019).

Ultimately, more work remains to be done to better understand what constitutes impact in

HIPs, for whom, and under what circumstances. When such inquiry is undertaken using

statistical methods that account for factors such as selection bias, diversity in the student

population, and range of choices across the undergraduate experience while interrogating

outcomes that target student learning, the prospect of uncovering a nuanced understanding of the

impact of participation in HIPs is enhanced as is the quality of actionable insights that can inform
 139

decision making for current and future college students as well as the postsecondary institutions

at which they enroll.

 140

CHAPTER VI

CONCLUSION

The methods and results of these analyses extend the scholarly discourse on the effects of

participation in HIPs on college outcomes with emphases on the mitigation of selection bias,

effects of participation in two or more HIPs as recommended by Kuh (2008), and exploration of

outcomes that extend beyond the conclusion of undergraduate enrollment. In doing so, the

present study likewise addresses some of the longstanding limitations of the HIP scholarship

(Brownell & Swaner, 2010) via use of a multi-institutional dataset and examination of multiple

types of HIPs. Via these multiple extensions of the current literature base, the present study

offers insights to researchers, institutions, and students that can inform future inquiry as well as

program design and decision making.

First, the use of PS analysis with SEM (Leite et al., 2019) in the present study offers

evidence to support the inclusion of such approaches in the design of future studies. The ability

to account for student choice via self-selection into HIP experiences is well-aligned with theories

of college outcomes (Astin, 1984) as is the inclusion of a diverse set of covariates from multiple

domains (Reason, 2009). The use of PS analysis to balance group differences independent of the

post-graduation college outcomes also provides a means to further reduce the potential for error

(Steiner et al., 2010) as does the use of doubly robust PS estimation (Bang & Robins, 2005;

Schafer & Kang, 2008). The use of SEM further permits the inclusion of latent variables,

expanding the nature and complexity of prospective outcomes that can be explored (Schumacker

& Lomax, 2016). Thus, the methodology employed in the present study expands the range of

established and tested analytic techniques available to researchers, thereby permitting more
 141

rigorous statistical analysis to enhance the clarity of conclusions that can be drawn regarding

benefits to participation in HIP experiences.

Additionally, the results of the present study provide insight into the influence of

participation in at least two HIP experiences, a longstanding recommendation (Kuh, 2008) that

has been less frequently examined in the literature. In particular, the results described in

Manuscript I (see Chapter 4) offer evidence of benefit to participation in two or more HIP

experiences with a statistically significant main effect as well as positive and statistically

significant differences in perceived importance of postsecondary education in preparation for

adult life for female students, students from low SES backgrounds, and students who are

members of minoritized racial/ethnic populations. These findings signal particular benefit from

participation in two or more HIPs for certain student populations that can aid institutions in

program design and students in decision making. Similarly, the results described in Manuscript II

(see Chapter 5) likewise offer important insights to institutions and students. The lack of

statistical significance in both the main and interaction effects suggests that participation in two

or more HIPs is insufficient to influence perceived learning and challenge in professional

experiences. While HIP experiences foreground learning and challenge via sustained

independent inquiry, student interaction with faculty and peers, and regular feedback, synthesis,

and perspective taking (Kuh, 2008), there are likely other variables that might explain why

students choose professional experiences in which they perceive higher greater levels of learning

and challenge. Institutions and students who endeavor to develop attitudes and skills related to

lifelong learning may require additional learning, practice, and support in these domains.
 142

Finally, the results of the present study provide evidence that signals the post-graduation

outcomes that might be influenced by participation in HIP experiences as well as those that

might not be influenced by such participation. Since much of the foundational and contemporary

HIP scholarship focused on college outcomes that occur either while students are enrolled or at

the conclusion of their enrollment, the ability to consider post-graduation benefits to

participation in HIP experiences reflects the reality that postsecondary educational experiences

are not ends unto themselves but rather designed to prepare students for adulthood as well as a

myriad of personal and professional endeavors. In this regard, the evidence of positive and

statistically significant differences in perceived importance of postsecondary education in

preparation for adult life for female students, students from low SES backgrounds, and students

who are members of minoritized racial/ethnic populations signals an additional and extended

value proposition for participation in HIP experiences, particularly for students who have not

historically enjoyed advantages in terms of postsecondary access. Conversely, the absence of

statistically significant findings related to perceived learning and challenge in professional

experiences suggests that additional learning, practice, and institutional support that extends

beyond the ability to participate in HIP experiences may be needed to provide students with the

ability to seek out, attain, and retain professional experiences that prioritize learning and

challenge in the same way that HIP experiences themselves are designed to do during

undergraduate enrollment.
APPENDICES
 APPENDIX A

NSSE SCALE OF EDUCATIONALLY PURPOSEFUL ACTIVITIES

 145

Appendix A
NSSE Scale of Educationally Purposeful Activities

(Kuh, Kinzie, Cruce, et al., 2007)

• Asked questions in class or contributed to class discussion

• Made a class presentation
• Prepared for two or more drafts of a paper or assignment before turning it in
• Come to class without completing readings or assignments
• Worked with other students on projects during class
• Worked with classmates outside of class to prepare class assignments
• Tutored or taught other students (paid or voluntary)
• Participated in a community-based project as part of a regular course
• Used an electronic medium (list-serv, chat group, Internet, etc.) to discuss or
complete an assignment
• Used email to communicate with an instructor
• Discussed grades or assignments with an instructor
• Talked about career plans with a faculty member or advisor
• Discussed ideas from your readings or classes with faculty members outside of
class
• Received prompt feedback from faculty on your academic performance (written
or oral)
• Worked harder than you thought you could to meet an instructor’s demands
• Worked with faculty members on activities other than coursework (committees,
orientation student life activities, etc.)
• Discussed ideas from your readings or classes with others outside of class
(students, faculty members, coworkers, etc.)
• Had serious conversations with students of a different race or ethnicity than your
own
• Had serious conversations with students who differ from you in terms of their
religious beliefs, political opinions, or personal values

2000 NSSE Response Scale: “Very Often,” “Often,” “Occasionally,” “Never”

2001-2003 NSSE Response Scale: “Very Often,” “Often,” “Sometimes,” “Never”
 APPENDIX B

LIST OF MODEL COVARIATES

 147
Appendix B
List of Model Covariates
Covariate Description ELS Variable(s) ELS Wave
Student Precollege Characteristics and Experiences
Sociodemographic traits
SCSD-RACE Membership in a minoritized racial population F1RACE 2004
SCSD-SEX Biological sex F1SEX 2004
SCSD-SES SES F1SES2 2004
SCSD-FAMSUP Perception of support network aspirations F1S44A-F1S44D 2004

Academic performance and preparation

SCHS-GPA High school GPA F1RGPP2 2004
SCHS-PREP Perception of academic preparedness F2B17A-F2B1C 2006

Student dispositions
SCD-GOAL Educational expectations F1STEXP 2004

Organizational Context
ORG1-CON Control of first postsecondary institution F3PS1SEC 2012
ORG1-LEV Level of first postsecondary institution F3PS1SEC 2012

Individual Student Experiences

Curricular experiences
EXPCU-TSFR Student respondent’s transfer status F3PS1RETAIN 2012

Field of study for first known bachelor’s degree

EXPCU-ART Arts and humanities F3TZBCH1CIP2 2012
EXPCU-BUS Business F3TZBCH1CIP2 2012
EXPCU-EDU Education F3TZBCH1CIP2 2012
EXPCU-HHS Health and human services F3TZBCH1CIP2 2012
EXPCU-SS Social science F3TZBCH1CIP2 2012
EXPCU-STEM Science, technology, engineering, and math F3TZBCH1CIP2 2012
(STEM)

Out-of-class experiences
EXPCO-FAC Talk with faculty outside of class F2B18A 2006
EXPCO-ADV Meet with their advisor about academic plans F2B18B 2006
EXPCO-INT Participate in intramural or nonvarsity sports F2B18E 2006
EXPCO-VAR Participate in varsity or intercollegiate sports F2B18F 2006
EXPCO-EXT Participate in extracurricular activities F2B18G 2006
EXPCO-JOB1 Number of jobs while enrolled in 2004-2005 F2C24_P 2006
EXPCO-JOB2 Number of jobs while enrolled in 2005-2006 F2C29_P 2006

EXPCO-MEN Access to mentoring experience as F3A14F 2012

undergraduate
 APPENDIX C
CLASSIFICATION OF INSTRUCTIONAL PROGRAMS (CIP) CODE AGGREGATION

Appendix C
Classification of Instructional Programs (CIP) Code Aggregation
Category
Arts & Humanities (EXPCU-ART)
Business (EXPCU-BUS)
Education (EXPCU-EDU)
Health and Human Services (EXPCU-
HHS)
Social Sciences (EXPCU-SS)
Science, Technology, Engineering, and
Math (STEM) (EXPCU-STEM)
149
Constituent CIP Codes as designated by NCES
Architecture and related services (4)
Area, Ethnic, Cultural, and Gender Studies (5)
Foreign languages/literature/linguistics (16)
Family/consumer sciences/human sciences (19)
English language and literature/letters (23)
Liberal arts/science/general studies/humanities (24)
Philosophy and religious studies (38)
Theology and religious vocations (39)
Visual and performing arts (50)
Business/management/marketing/related (52)
Education (13)
Parks/recreation/leisure/fitness studies (31)
Public administration/social service (44)
Health/related clinical sciences (51)
Communication, journalism, related (9)
Social sciences (45)
Psychology (42)
History (54)
Agriculture/operations/related sciences (1)
Natural resources and conservation (3)
Computer/information science/support (11)
Engineering (14)
Engineering technologies/technicians (15)
Biological and biomedical sciences (26)
Mathematics and statistics (27)
Physical sciences (40)
 APPENDIX D

CONCEPTUAL MODEL FOR PS ESTIMATION

 151
Appendix D
Conceptual Model for PS Estimation
 APPENDIX E

CONCEPTUAL MODEL FOR MANUSCRIPT I MULTIPLE-GROUP SEM

 153
Appendix E
Conceptual Model for Manuscript I Multiple-Group SEM
 APPENDIX F

CONCEPTUAL MODEL FOR MANUSCRIPT II MULTIPLE-GROUP SEM

 155

Appendix F
Conceptual Model for Manuscript II Multiple-Group SEM
REFERENCES
 157

REFERENCES

ACT. (2007). The role of nonacademic factors in college readiness and success. ACT.

https://www.act.org/content/dam/act/unsecured/documents/nonacademic_factors.pdf

Adelman, C. (2006). The toolbox revisited: Paths to degree completion from high school

through college. Washington, DC: U.S. Department of Education.

https://www.researchgate.net/publication/275463367_The_Toolbox_Revisited_Paths_

to_Degree_Completion_From_High_School_Through_College

Allensworth, E. M., & Clark, K. (2020). High school GPAs and ACT scores as predictors of

college completion: Examining assumptions about consistency across high schools.

Educational Researcher, 49(3), 198-211. https://doi.org/10.3102/0013189X20902110

Allison, P.D. (2002). Missing data. Sage Publications. https://doi.org/10.4135/

9781412985079

Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How

learning works: Seven research-based principles for smart teaching. John Wiley &

Sons.

Asparouhov, T., Muthén, B., & Muthén, B. O. (2006). Robust chi square difference testing

with mean and variance adjusted test statistics. https://www.statmodel.com/download/

webnotes/webnote10.pdf

Astin, A. W. (1984). Student involvement: A developmental theory for higher education.

Journal of College Student Personnel, 25(4), 297-308. https://www.researchgate.net/

publication/220017441_Student_Involvement_A_Development_Theory_for_Higher_

Education
 158

Astin, A.W. (1993). What matters in college? Four critical years revisited. Jossey-Bass.

Association of American Colleges and Universities (2007). College learning for the new

global century: A report from the National Leadership Council for Liberal

Education and America’s Promise. https://www.aacu.org/sites/default/files/files

/LEAP/GlobalCentury_final.pdf

Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects

of confounding in observational studies. Multivariate Behavioral Research, 46(3),

399-424. https://doi.org/10.1080/00273171.2011.568786

Bang, H., & Robins, J. M. (2005). Doubly robust estimation in missing data and causal

inference models. Biometrics, 61(4), 962-973. https://doi.org/10.1111/j.1541-

0420.2005.00377.x

Barr, A., & Turner, S. E. (2013). Expanding enrollments and contracting state budgets: The

effect of the Great Recession on higher education. The Annals of the American

Academy of Political and Social Science, 650(1), 168-193. https://doi.org/

10.1177/0002716213500035

Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood

versus means and variance adjusted weighted least squares estimation in CFA.

Structural Equation Modeling, 13(2), 186-203. https://doi.org/10.1207/

s15328007sem1302_2

Brown, S. D., Tramayne, S., Hoxha, D., Telander, K., Fan, X., & Lent, R. W. (2008).

Social cognitive predictors of college students’ academic performance and

persistence: A meta-analytic path analysis. Journal of Vocational Behavior, 72(3),

 159

298-308. https://doi.org/10.1016/j.jvb.2007.09.003

Brownell, J. E., & Swaner, L. E. (2010). Five high-impact practices: Research on learning

outcomes, completion, and quality. Association of American Colleges and

Universities.

Bowman, N. A., & Holmes, J. M. (2018). Getting off to a good start? First-year

undergraduate research experiences and student outcomes. Higher Education, 76(1),

17-33. https://doi.org/10.1007/s10734-017-0191-4

Cheema, J. R. (2014). Some general guidelines for choosing missing data handling methods

in educational research. Journal of Modern Applied Statistical Methods, 13(2), 53-

75. https://doi.org/10.22237/jmasm/1414814520

Cox, B. E., McIntosh, K., Reason, R. D., & Terenzini, P. T. (2014). Working with missing

data in higher education research: A primer and real-world example. The Review of

Higher Education, 37(3), 377-402. https://doi.org/10.1353/rhe.2014.0026

Culver, K. C., & Bowman, N. A. (2020). Is what glitters really gold? A quasi-experimental

study of first-year seminars and college student success. Research in Higher

Education, 61(2), 167-196. https://doi.org/10.1007/s11162-019-09558-8

Enders, C. K., & Mansolf, M. (2018). Assessing the fit of structural equation models with

multiply imputed data. Psychological Methods, 23(1), 76. https://doi.org/10.1037/

met0000102

Espinosa, L. L., Turk, J. M., Taylor, M., & Chessman, H. M. (2019). Race and ethnicity in

higher education: A status report. American Council on Education.

https://1xfsu31b52d33idlp13twtos-wpengine.netdna-ssl.com/wp-
 160

content/uploads/2019/02/Race-and-Ethnicity-in-Higher-Education.pdf

Finley, A. (2021). How college contributes to workforce success. Association of American

Colleges and Universities. https://www.aacu.org/sites/default/files/files/research/

AACUEmployerReport2021.pdf

Finley, A., & McNair, T. (2013). Assessing underserved students’ engagement in high-

impact practices. Association of American Colleges and Universities.

https://www.aacu.org/sites/default/files/files/assessinghips/AssessingHIPS_TGGran

tReport.pdf

Friga, N. (2021, February 5). How much has Covid cost colleges? $183 billion. The

Chronicle of Higher Education. https://www.chronicle.com/article/how-to-fight-

covids-financial-crush

Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual

Review of Psychology, 60, 549-576. https://doi.org/10.1146/annurev.psych.

58.110405.085530

Harring, J. R., McNeish, D. M., & Hancock, G. R. (2017). Using phantom variables in

structural equation modeling to assess model sensitivity to external

misspecification. Psychological Methods, 22(4), 616. https://doi.org/10.1037/

met0000103

Hart Research Associates. (2015). Falling short? College learning and career success.

Association of American Colleges and Universities. https://www.aacu.org/

sites/default/files/files/LEAP/2015employerstudentsurvey.pdf

Hart Research Associates. (2018). Fulfilling the American dream: Liberal education and
 161

the future of work. Association of American Colleges and Universities.

https://www.aacu.org/sites/default/files/files/LEAP/2018EmployerResearchReport.

pdf

Heckman, J. J. (2005). The scientific model of causality. Sociological Methodology, 35(1),

1-97. https://doi.org/10.1111/j.0081-1750.2006.00164.x

Hill, J. (2004). Reducing bias in treatment effect estimation in observational studies

suffering from missing data. Institute for Social and Economic Research and Policy.

https://academiccommons.columbia.edu/doi/10.7916/D8B85G11

Ingels, S. J., Pratt, D. J., Rogers, J. E., Siegel, P. H., & Stutts, E. S. (2004). Education

Longitudinal Study of 2002: Base Year Data File User's Manual. NCES 2004-

405. National Center for Education Statistics. https://nces.ed.gov/pubs2004/

2004405.pdf

Ingels, S. J., Pratt, D. J., Rogers, J. E., Siegel, P. H., & Stutts, E. S. (2005). Education

Longitudinal Study of 2002: Base-Year to First Follow-Up Data File

Documentation. NCES 2006-344. National Center for Education Statistics.

https://nces.ed.gov/pubs2006/2006344.pdf

Ingels, S. J., Pratt, D. J., Wilson, D., Burns, L. J., Currivan, D., Rogers, J. E., & Hubbard-

Bednasz, S. (2007). Education Longitudinal Study of 2002 (ELS: 2002): Base-

Year to Second Follow-Up Data File Documentation. NCES 2008-347. National

Center for Education Statistics. https://nces.ed.gov/pubs2008/2008347.pdf

Ingels, S. J., Pratt, D. J., Alexander, C. P., Jewell, D. M., Lauff, E., Mattox, T. L., &

Wilson, D. (2014). Education Longitudinal Study of 2002 (ELS: 2002) Third

 162

Follow-up Data File Documentation. NCES 2014-364. National Center for

Education Statistics. https://nces.ed.gov/pubs2014/2014364.pdf

Johnson, S. R., & Stage, F. K. (2018). Academic engagement and student success: Do

high- impact practices mean higher graduation rates? The Journal of Higher

Education, 1-29. https://doi.org/10.1080/00221546.2018.1441107

Klebs, S., Nguyen, S., Fishman, R., & Hiler, T. (2021, June 29). One year later: COVID-

19’s impact on current and future college students. Third Way.

https://www.thirdway.org/memo/one-year-later-covid-19s-impact-on-current-and-

future-college-students

Kilgo, C. A., Sheets, J. K. E., & Pascarella, E. T. (2015). The link between high-impact

practices and student learning: Some longitudinal evidence. Higher Education,

69(4), 509-525. https://doi.org/10.1007/s10734-014-9788-z

Kinzie, J., Gonyea, R., Shoup, R., & Kuh, G. D. (2008). Promoting persistence and success

of underrepresented students: Lessons for teaching and learning. New Directions for

Teaching and Learning, 2008(115), 21-38. https://doi.org/10.1002/tl.323

Kuh, G.D. (2008). High-impact educational practices: What they are, who has access to

them, and why they matter. Association of American Colleges and Universities.

Kuh, G. D., Cruce, T. M., Shoup, R., Kinzie, J., & Gonyea, R. M. (2008). Unmasking the

effects of student engagement on first-year college grades and persistence. The

Journal of Higher Education, 79(5), 540-563. https://doi.org/10.1080/

00221546.2008.11772116

Kuh, G. D., & Kinzie, J. (2018). What really makes a “high-impact” practice high impact.
 163

Inside Higher Ed. https://www.insidehighered.com/views/2018/05/01/kuh-and-

kinzie-respond-essay-questioning-high-impact-practices-opinion

Kuh, G.D., Kinzie, J., Buckley, J.A., Bridges, B.K. & Hayeck, J.C. (2007). Piecing together

the student success puzzle: Research, propositions, and recommendations. ASHE

Higher Education Report, 32(5). https://doi.org/10.1353/rhe.2007.0063

Kuh, G. D., Kinzie, J., Cruce, T., Shoup, R., & Gonyea, R. M. (2007). Connecting the dots:

Multi-faceted analyses of the relationships between student engagement results

from the NSSE, and the institutional practices and conditions that foster student

success. Center for Postsecondary Research. http://hdl.handle.net/2022/23684

Laird, T. F. N., Shoup, R., Kuh, G. D., & Schwarz, M. J. (2008). The effects of discipline

on deep approaches to student learning and college outcomes. Research in Higher

Education, 49(6), 469-494. https://doi.org/10.1007/s11162-008-9088-5

Leite, W. L., Stapleton, L. M., & Bettini, E. F. (2019). Propensity score analysis of

complex survey data with structural equation modeling: A tutorial with MPlus.

Structural Equation Modeling: A Multidisciplinary Journal, 26(3), 448-469.

https://doi.org/10.1080/10705511.2018.1522591

Lombardi, A., Seburn, M., & Conley, D. (2011). Treatment of nonresponse items on scale

validation: What “don't know” responses indicate about college readiness.

https://files.eric.ed.gov/fulltext/ED518170.pdf.

McNeish, D., Stapleton, L. M., & Silverman, R. D. (2017). On the unnecessary ubiquity of

hierarchical linear modeling. Psychological Methods, 22(1), 114.

https://doi.org/10.1037/met0000078
 164

McDaniel, A., & Van Jura, M. (2020). High-impact practices: Evaluating their effect on

college completion. Journal of College Student Retention: Research, Theory &

Practice, 1-18. https://doi.org/10.1177/1521025120947357

McNair, T. B., Bensimon, E. M., & Malcom-Piqueux, L. (2020). From equity talk to equity

walk: Expanding practitioner knowledge for racial justice in higher education.

John Wiley & Sons. https://doi.org/10.1002/9781119428725

Miller, A. L., Rocconi, L. M., & Dumford, A. D. (2018). Focus on the finish line: does

high-impact practice participation influence career plans and early job attainment?

Higher Education, 75(3), 489-506. https://doi.org/10.1007/s10734-017-0151-z

Millsap, R. E., & Olivera‑Aguilar, M. (2012). Investigating measurement invariance using

confirmatory factor analysis. In R.H. Hoyle (Ed.), Handbook of Structural Equation

Modeling (pp. 380-392). Guilford Press.

Mitra, R., & Reiter, J. P. (2016). A comparison of two methods of estimating propensity

scores after multiple imputation. Statistical methods in medical research, 25(1),

188-204. https://doi.org/10.1177/0962280212445945

Muthén, L.K. & Muthén, B.O. (2017). MPlus user’s guide (8th ed.). Los Angeles, CA.

https://www.statmodel.com/ugexcerpts.shtml

Myers, C.B., Myers, S.M., and Peters, M. (2018). The longitudinal connections between

undergraduate high impact curriculum practices and civic engagement in adulthood.

Research in Higher Education, 60, 83-110. https://doi.org/10.1007/s11162-018-

9504-4

National Center for Education Statistics (n.d.). Education Longitudinal Study of 2002
 165

(ELS: 2002) online codebook [Data set]. https://nces.ed.gov/OnlineCodebook/

Session/Codebook/a111f850-113c-4da9-a394-fd3f212385e7.

National Study of Student Engagement (2001). Origins and potential of the National Study

for Student Engagement. https://nsse.indiana.edu/nsse/about-nsse/general-nsse-info-

facts/origins.html.

National Study of Student Engagement (2007). Experiences that matter: Enhancing student

learning and success. https://scholarworks.iu.edu/dspace/handle/2022/23412

National Study of Student Engagement (2021). NSSE topical modules.

https://nsse.indiana.edu/nsse/survey-instruments/topical-modules/index.html

Pascarella, E.T., & Terenzini, P.T. (2005). How college affects students: A third decade of

research (Vol. 2). Jossey-Bass.

Patton, L. D., Harper, S. R., & Harris, J. (2015). Using critical race theory to (re)interpret

widely studied topics related to students in US higher education. In A.M. Martínez

Alemán, E.M. Bensimon, & B. Pusser (Eds.), Critical Approaches to the Study of

Higher Education (pp. 193-219). Johns Hopkins University Press.

Provencher, A., & Kassel, R. (2019). High-impact practices and sophomore retention:

Examining the effects of selection bias. Journal of College Student Retention:

Research, Theory & Practice, 21(2), 221-241. https://doi.org/10.1177/

1521025117697728

Purdue Online Writing Lab. (n.d.). In-text citations: Author/authors. Purdue Online Writing

Lab. https://owl.purdue.edu/owl/research_and_citation/apa_style/apa_formatting

_and_style_guide/in_text_citations_author_authors.html
 166

Purdue Online Writing Lab. (n.d.). Reference list: Articles in periodicals. Purdue Online

Writing Lab. https://owl.purdue.edu/owl/research_and_citation/apa_style/apa_

formatting_and_style_guide/reference_list_articles_in_periodicals.html

Purdue Online Writing Lab. (n.d.). Reference list: Author/authors. Purdue Online Writing

Lab. https://owl.purdue.edu/owl/research_and_citation/apa_style/apa_formatting

_and_style_guide/reference_list_author_authors.html

Purdue Online Writing Lab. (n.d.). Reference list: Books. Purdue Online Writing Lab.

https://owl.purdue.edu/owl/research_and_citation/apa_style/apa_formatting_and_st

yle_guide/reference_list_books.html

Purdue Online Writing Lab. (n.d.). Electronic sources. Purdue Online Writing Lab.

https://owl.purdue.edu/owl/research_and_citation/apa_style/apa_formatting_and_st

yle_guide/reference_list_electronic_sources.html

Purdue Online Writing Lab. (n.d.). Reference list: Other print sources. Purdue Online

Writing Lab. https://owl.purdue.edu/owl/research_and_citation/apa_style/apa_

formatting_and_style_guide/reference_list_other_print_sources.html

Reason, R. D. (2003). Student variables that predict retention: Recent research and new

developments. Journal of Student Affairs Research and Practice, 40(4), 704-723.

Reason, R. D. (2009). An examination of persistence research through the lens of a

comprehensive conceptual framework. Journal of College Student Development,

50(6), 659-682. https://doi.org/10.2202/1949-6605.5022

Ro, H. K., Terenzini, P. T., & Yin, A. C. (2013). Between-college effects on students

reconsidered. Research in Higher Education, 54(3), 253-282. https://doi.org/

 167

10.1007/S11162-012-9269-0

Robbins, S. B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004). Do

psychosocial and study skill factors predict college outcomes? A meta-analysis.

Psychological Bulletin, 130(2), 261. https://doi.org/10.1037/0033-2909.130.2.261

Rosenbaum, P.R. (2002). Observational studies. Springer Publications.

Seifert, T. A., Gillig, B., Hanson, J. M., Pascarella, E. T., & Blaich, C. F. (2014). The

conditional nature of high impact/good practices on student learning outcomes. The

Journal of Higher Education, 85(4), 531-564. https://doi.org/10.1080/

00221546.2014.11777339

Schafer, J. L., & Kang, J. (2008). Average causal effects from nonrandomized studies: a

practical guide and simulated example. Psychological Methods, 13(4), 279.

https://doi.org/10.1037/a0014268

Schumacker, R. E., & Lomax, R. G. (2016). A beginner's guide to structural equation

modeling (4th Ed.). New York, NY: Routledge. https://doi.org/

10.4324/9781410610904

Stapleton, L. M. (2008). Variance estimation using replication methods in structural

equation modeling with complex sample data. Structural Equation Modeling: A

Multidisciplinary Journal, 15(2), 183-210. https://doi.org/10.1080/

10705510801922316

Steiner, P. M., Cook, T. D., Li, W., & Clark, M. H. (2015). Bias reduction in quasi-

experiments with little selection theory but many covariates. Journal of Research

on Educational Effectiveness, 8(4), 552-576. https://doi.org/10.1080/

 168

19345747.2014.978058

Steiner, P.M., Cook, T.D., Shadish, W.R., & Clark, M.H. (2010). The importance of

covariate selection in controlling for selection bias in observational studies.

Psychological Methods, 15(3). 250-267. https://doi.org/10.1037/a0018719

Streiner, D. L. (2006). Building a better model: an introduction to structural equation

modelling. The Canadian Journal of Psychiatry, 51(5), 317-324. https://doi.org/

10.1177/070674370605100507

Stuart, E. A., & Rubin, D. B. (2008). Best practices in quasi-experimental designs:

Matching methods for causal inference. In J. Osbourne (Ed.) Best practices in

quantitative methods (p. 155-176). Thousand Oaks, CA: Sage Publications.

https://doi.org/10.4135/9781412995627

Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward.

Statistical Science: A Review Journal of the Institute of Mathematical Statistics,

25(1), 1. https://doi.org/10.1214/09-STS313

Terenzini, P. T., & Reason, R. D. (2005, November). Parsing the first year of college:

Rethinking the effects of college on students. [Conference Paper]. Annual

Conference of the Association for the Study of Higher Education, Philadelphia, PA.

Terenzini, P. T., Springer, L., Yaeger, P. M., Pascarella, E. T., & Nora, A. (1996). First-

generation college students: Characteristics, experiences, and cognitive

development. Research in Higher Education, 37(1), 1-22.

https://www.jstor.org/stable/40196208

Thoemmes, F. J., & Kim, E. S. (2011). A systematic review of propensity score methods in
 169

the social sciences. Multivariate Behavioral Research, 46(1), 90-118.

https://doi.org/10.1080/00273171.2011.540475

Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd

ed.). University of Chicago Press.

Valentine, J., Price, D., & Yang, H. (2021). Lumina issue paper: High-impact practices

and gains in student learning: Evidence from Georgia, Montana, and Wisconsin.

Lumina Foundation. http://nashonline.org/wp-content/uploads/2021/03/high-

impact-practices-and-gains-in-student-learning.pdf

Wang, J., & Wang, X. (2019). Structural equation modeling: Applications using MPlus.

John Wiley & Sons. https://doi.org/10.1002/9781118356258

Wolniak, G. C., & Engberg, M. E. (2019). Do “high-impact” college experiences affect

early career outcomes? The Review of Higher Education, 42(3), 825-858.

https://doi.org/10.1353/rhe.2019.0021

Zilvinskis, J. (2019). Measuring quality in high-impact practices. Higher Education, 78(4),

687-709. https://doi.org/10.1007/s10734-019-00365-9