The Assessment of Intrapersonal and

Interpersonal Competencies

National Academy of Sciences

April 13, 2017

Wayne Camara
Horace Mann Research Chair and Senior Vice
President of Research
Uses of Intrapersonal and Interpersonal Assessments
1. Admissions and Selection
2. Placement (Developmental vs Credit bearing
courses)
3. Personal Growth and Formative Improvement
a) Proactive/Preparatory (K-12)
b) Proactive/Preparatory (prior to enrollment)
c) Remedial
4. Research
5. Accountability in Higher Education

2
Assessments should be driven by intended use

1. Intended purpose of assessment and

claims regarding score interpretations
2. Criteria (G&T, academic success,
engagement, ability to benefit, retention)
3. Relevant context or moderators
(commuting or residing at college, adult
student)
4. Substance (content) and method (format)
of assessment

3
Previous Efforts to Examine Intra- and Interpersonal
Constructs in Higher Education

4
ACT Holistic Framework (Camara et al, 2015)
• English Language Arts
Core Academic • Mathematics
Skills • Science

• Technology and Information Literacy

Cross-Cutting • Collaborative Problem Solving
Capabilities •
•
Studying and Learning
Thinking and Metacognition

• Acting Honestly Getting Along Well with Others

Behavioral Skills • Maintaining Composure Sustaining Effort
• Socializing with Others Keeping an Open Mind

• Self Knowledge
Education & • Environmental Factors
• Integration
Career Navigation • Managing Career and Education Actions

5
5
Holistic Model of College Success
Science
ELA
Math

Critical Thinking
Technology and Information
Literacy
Studying and Learning
College GPA
Persistence
Dependability
Self-Confidence

Socialization
Academic Self-Efficacy
Goals

6
Holistic Model of College Success
Science
ELA
Math

Critical Thinking
Technology and Info. Literacy
Studying and Learning
College
Persistence Graduation
Goal Striving
Socialability, Optimism
(Dependability, Self-
Confidence)
Academic Self-Efficacy
Fit, Supports
(Socialization
Academic Self-Efficacy)
Goals 7
Behavioral Framework Illustration

8
Growth in Admission Testing
ACT Administrations
4.0
3.5
3.0
2.5
Millions

2.0
1.5
1.0
0.5
0.0
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Total National State

• In 2014, 57% of high school graduates took the ACT.

9
Trends in Admissions Factors Considered
Important or Very Important

10
10
Measuring Intrapersonal and Interpersonal Constructs

A personal statement is • Rater biases

to college admissions as • Lack of verification of author
an unstructured interview • Susceptible to external
is to personnel selection. assistance
• Lack of standardized scoring
Comprehensive review is systems or rubrics for evaluation
used to evaluate the • ‘Context’ for school quality,
whole person in context individual’s background,
overcoming adversity is largely
and come to a summary
left to one’s own judgment
judgement.
• Absence of research on college
decision processes.
• Missing variables (test, AP, IB)
11
Assessments
Existing Measures in Higher Ed New Techniques
• Engage (6-8, 9-12,
• Anchoring Vignettes
College transition)
• Forced Choice / Ipsative
• Success Navigator
method
(College transition)
• Natural language
• Personal Potential Index –
processing – AI
PPI (graduate school)
• Gaming
• Motivated Strategies for
Learning Questionnaire • Big Data - aggregation
(College students)
• LASSI
• Individual institutions
practices
12
Validity, Reliability, Fairness
• Construct terms and definitions create confusion.
– Some terms suggest personality traits long considered to be
static. Other terms raise sensitivities about privacy especially in
public education.
• Evidence related to other variables may be most
essential type of validity for higher education adoption.
• Content evidence also essential – fidelity with what is
required to succeed in college (job analysis
methodology)
• Contextual factors – culture, climate, institution,
department.
– Difficult to transport validity evidence globally (teamwork,
sociability have different values in different cultures).

13
STEM
• Academic preparation is essential for STEM success –
but not determinative (Radunzel, Mattern, & Westrick,
2016)
– Many academically prepared students do not complete a STEM
degree – 47%
– While some students not academically prepared do complete a
STEM degree – 19%
• The highest STEM persistence and degree completion
rates are for students with both expressed and
measured STEM interests.
• The lowest rates were for students who had neither
expressed or measured interests in STEM.

14
Impediments in K-12
• Large scale assessments in K-12 today are driven
by accountability (not student centric) – evaluate
schools, teaching, educators.
• Intra- and Interpersonal constructs can not be
easily associated with a course, a teacher, or a
curriculum.
• Not ‘actively’ taught – worst yet, in some
communities constructs may be viewed as outside
the responsibility of a school (home factors).
• Items may be viewed as invasive and not relevant
to schools. (Contrast with career interests)
15
Impediments in Higher Education
• Demand is absent
– Additional hurdles may discourage applicants
– Gaining consensus on constructs, definitions, measures is
difficult
• Change is difficult
– Writing introduced in 2005 by ACT + SAT yet 80% of
colleges do not require that test.
• Admissions professionals feel they can assess these factors
with subjective measures – anecdotal evidence.
• Concern after legal rulings (Fisher v University of Texas,
Grantz v. Bollinger, Grutter v. Bollinger)
• The ‘criterion problem’ in higher education has largely
suppressed large scale research concerning new
predictors for college admissions (Camara, 2005).
16
Impediments to Research and Test Development
• Lack of demand by Higher Education – little
incentive for investment
• Institutional fit and Individual context
• Need operational research as well as experimental
studies
• Cross-institutional research collaboratives required
• Difference between experimental work and large
scale implementation
• Susceptibility of faking – overstated
• Similar acceptance issues as K-12, especially for
public institutions
17
On the positive side…

• Incremental validity
• Assess important outcomes – retention,
graduation, time to degree, engagement.
• Assess important constructs in demand by
employers – problem solving, teamwork,
technology literacy
• May reduce adverse impact against
underserved groups
• Provide a more holistic framework for
decisions – multiple measures.
18
Construct representation - Triathlon
• Triathlon’s have three events.
• Athletes often excel at different events.
• When you reduce a competition to just
2 of the 3 events it may result in
significantly different finishes!
• Ken finishes 1st in swimming, 5th in
running and 15th in biking. Overall he
would rank 7th. If only the first two
events are counted he would rank 3rd,
but if the last two events are counted
he would rank 10th.
• Similar to selecting a subset of relevant
constructs to focus on you may
inadvertently introduce construct
underrepresentation and bias.

19
Incremental Validity and Adverse Impact
Table 1. Correlations with First-Year GPA
(FYGPA)
Predictor FYGPA
Research based on ACT ACT 0.42
Engage shows that Engage Academic Discipline 0.27
Intrapersonal and ACT + Academic Discipline 0.51

Interpersonal factors,
particularly academic
Table 2. Subgroup Differences on ACT versus Engage
discipline: Academic Discipline
– Predicts important Engage Academic
outcomes ACT Discipline FYGPA
Subgroup M SD M SD M SD
– Has smaller subgroup
White 21.83 4.20 47.00 0.93 2.75 2.75
differences African American 17.72 3.32 49.00 0.95 2.30 2.30
Cohen's d 1.09 -0.26 0.48

20
Subgroup Impact at Differential Levels of Selectivity

Research based on ACT

Engage shows that ACT and Engage
Intrapersonal and Institutional
Selectivity Race/ethnicity ACT only
Academic
Discipline
Interpersonal factors, African American
American Indian
3.2
0.8
3.6
1.0
particularly academic Asian American 2.4 2.5
Hispanic 2.1 2.5
discipline: High Selectivity Pacific Islander - -
(top 15%) White 91.4 90.4
– Predicts important African American 9.4 11.2
outcomes American Indian
Asian American
1.0
2.1
1.1
2.1
– Has smaller subgroup Moderate Hispanic 3.3 3.8
Selectivity (top Pacific Islander 0.0 0.0
differences 50%) White 84.1 81.8
– Can increase diversity of African American
American Indian
14.3
1.1
16.9
1.1
admitted students Asian American 1.9 1.9
Hispanic 4.7 5.3
Low Selectivity Pacific Islander 0.0 0.0
(top 85%) White 78.0 74.8

21
 What we know vs what we do
• We know that performance in jobs and
in school (graduation, retention, grades,
performance) is influenced by multiple
factors…academic, behavioral, etc.
• We know that employers value
behaviors and use high stakes
assessments to measure these skills.
• We know that reading can confound
measurement of other constructs
(math, science and behaviors) and
introduce construct irrelevance resulting
in adverse impact.
• We know that academic factors have the
largest subgroup differences.
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). All other trademarks are property of their respective owners. 36655.
• But educational assessments focus on
academic skills such as math and ELA,
ignoring other important constructs.
• But in education behaviors are rarely
incorporated in standardized assessment
systems and may be evaluated in
admissions by individual raters using
subjective processes.
• But the common core has resulted in math
and science tests with heavy reading loads
and when behavioral skills are assessed
they are often measured in written formats
as opposed to simulations, SJTs (as in
industrial testing).
• But we use academic assessments as the
primary or sole focus in accountability,
admissions, and achievement.
22
Intrapersonal and Interpersonal Assessments for
Admissions
There are differences in access, resources, and experiences in
and out of school between ethnic groups and families which
explain roughly

There is a decision to measure some constructs (academic)

and not other constructs (behavior, cross-cutting skills)

The constructs we measure have larger adverse impact against

minorities, first-generation students, and low-income students
than factors we do not measure.

Institutions state they value the these other constructs and

consider them in their decisions – but, they are not measured
with standardized and reliable measures, nor part of the
systematic assessment system.
23