You are on page 1of 132

Judgment

Assessment Technical Manual


Hogan Judgment Assessment
Technical Manual

Hogan Assessment Systems


Tulsa, OK 74114, USA

2014


2014 Hogan Assessment Systems, Inc.

No part of this work may be copied or transferred to any other form or expression without
the expressed written consent of Hogan Assessment Systems, Inc.

Hogan Personality Inventory

Hogan Development Survey

Motives, Values, Preferences Inventory

Are exclusive registered trademarks of Hogan Assessment Systems, Inc.

hoganassessments.com

978-0-9889286-9-5


CONTENTS

CONTENTS

1. Conceptual Background 1

1.1 Introduction 1

1.2 Heuristics, Biases, and Rationalization 1

1.3 Judgment and Leadership 3

1.4 Current State of Judgment and Decision-Making Research 4

1.5 The Hogan Judgment Model 5

1.5.1 Information Processing 5

1.5.2 Using Personality to Assess Decision-Making Approaches 6

1.5.2.1 Threat Avoidance vs. Reward Seeking 7

1.5.2.2 Tactical vs. Strategic Thinking 7

1.5.2.3 Data-Driven vs. Intuitive Decisions 8

1.5.3 Using Personality to Assess Reactions to Feedback 8

1.5.3.1 Defensive vs. Cool-Headed 9

1.5.3.2 Denial vs. Acceptance 9

1.5.3.3 Superficial vs. Genuine Engagement 10

2. Inventory Construction, Reliability, and Structural Psychometrics 11

2.1 Development 11

2.1.1 Information Processing 11

2.1.1.1 Numerical Information Processing 12

2.1.1.2 Verbal Information Processing 12

2.1.2 Decision-Making Approaches & Decision-Making Styles 13

2.1.3 Reactions to Feedback & Openness to Feedback and Coaching 13

2.2 Definitions of the Scales 14

2.3 Item Composition of the Scales 16

III

CONTENTS

2.3.1 Numerical Information Processing 16

2.3.2 Verbal Information Processing 16

2.3.3 Decision-Making Approaches and Reactions to Feedback 16

2.4 Descriptive Statistics and Reliabilities 17

2.5 Parallel Forms Evidence for Numerical and Verbal Scales 19

2.6 Intercorrelations Between Hogan Judgment Model Scales 20

2.7 Structural Psychometrics of the Pre- and Post-Decision Scales 21

2.7.1 Pre-Decision Scales 22

2.7.2 Post-Decision Scales 23

3. Validity 24

3.1 Construct Validity 24

3.1.1 Procedure 24

3.1.2 Samples and Instruments 24

3.1.3 Results of Scale to Scale Correlates 31

3.2 Correlations with Others Descriptions 40

3.2.1 Procedure 40

3.2.2 Samples and Instruments 40

3.2.3 Results of Scale and Observer Description Correlates 41

3.3 Criterion-Related Validity 46

3.3.1 Procedure 46

3.3.2 Sample and Instrument 46

4. Interpretation and Uses 48

4.1 Information Processing 48

4.1.1 Verbal Information Processing 48

4.1.2 Numerical Information Processing 49

IV

CONTENTS

4.1.3 Information Processing Styles 49

4.1.3.1 Deliberate 49

4.1.3.2 Quantitative 50

4.1.3.3 Qualitative 50

4.1.3.4 Versatile 50

4.2 Decision-Making Approaches 50

4.2.1 Threat Avoidance vs. Reward Seeking 51

4.2.1.1 Developmental Recommendations for Threat Avoidance 52

4.2.1.2 Developmental Recommendations for Reward Seeking 52

4.2.2 Tactical vs. Strategic Thinking 53

4.2.2.1 Developmental Recommendations for Tactical Thinkers 54

4.2.2.2 Developmental Recommendations for Strategic Thinkers 54

4.2.3 Data-Driven vs. Intuitive Decisions 55

4.2.3.1 Developmental Recommendations for Data-Driven Decisions 56

4.2.3.2 Developmental Recommendations for Intuitive Decisions 56

4.3 Decision-Making Styles 57

4.3.1 Auditors 57

4.3.2 Surgeons 58

4.3.3 Stock Traders 58

4.3.4 Defense Analysts 58

4.3.5 Politicians 58

4.3.6 Chess Players 58

4.3.7 Promoters 58

4.3.8 Investors 59

4.4 Reactions to Feedback 59

V

CONTENTS

4.4.1 Defensive vs. Cool-Headed 59

4.4.2 Denial vs. Acceptance 60

4.4.3 Superficial vs. Genuine Engagement 60

4.5 Feedback Response Patterns 60

4.5.1 Defensive, Denial, Superficial 61

4.5.2 Defensive, Acceptance, Superficial 61

4.5.3 Defensive, Denial, Genuine 61

4.5.4 Defensive, Acceptance, Genuine 61

4.5.5 Cool-Headed, Denial, Superficial 61

4.5.6 Cool-Headed, Denial, Genuine 62

4.5.7 Cool-Headed, Acceptance, Superficial 62

4.5.8 Cool-Headed, Acceptance, Genuine 62

4.6 Openness to Feedback and Coaching 62

4.6.1 Resistant 63

4.6.2 Neutral 63

4.6.3 Receptive 63

5. Administering the Hogan Judgment Report 65

5.1 Key Features of the Online Assessment 65

5.2 Completing the Assessment Using the Online Platform 65

5.2.1 Numerical Section 68

5.2.2 Verbal Section 73

5.2.3 Decision-Making Style Section 78

5.3 Participant Informed Consent 82

5.3.1 Consent 83

5.3.2 Purpose 83

VI

CONTENTS

5.3.3 Data Processing 83

5.3.4 Access to Data 83

5.3.5 Security 84

5.3.6 Contact 84

5.4 Accommodating Individuals with Disabilities 84

5.5 Frequently Asked Questions 84

REFERENCES 86

APPENDIX A: Sample Hogan Judgment Report 98

APPENDIX B: Complete Correlation Matrices 107

VII

TABLES & FIGURES

TABLES & FIGURES



Figure 1.1 Hogan Judgment Model ................................................................................... 5

Table 2.1 Classical Scale Statistics for Hogan Judgment Model Scales ......... 18

Table 2.2 Parallel Forms Evidence for Numerical Information Processing ..... 19

Table 2.3 Parallel Forms Evidence for Verbal Information Processing ............ 20

Table 2.4 Correlations between Hogan Judgment Model Scales ........................ 20

Table 2.5 Sample Ethnic Composition by Age and Gender ................................... 21

Table 2.6 Varimax Rotated Factor Matrix for Pre-Decision Scales .................... 22

Table 2.7 Varimax Rotated Factor Matrix for Post-Decision Scales .................. 23

Table 3.1 Construct Validity Evidence for Threat Avoidance vs. Reward
Seeking Scale .......................................................................................................................... 31

Table 3.2 Construct Validity Evidence for Tactical vs. Strategic Thinking
Scale ............................................................................................................................................ 33

Table 3.3 Construct Validity Evidence for Data-Driven vs. Intuitive Decisions
Scale ............................................................................................................................................ 34

Table 3.4 Construct Validity Evidence for Defensive vs. Cool-Headed Scale . 35

Table 3.5 Construct Validity Evidence for Denial vs. Acceptance Scale .......... 36

Table 3.6 Construct Validity Evidence for Superficial vs. Genuine


Engagement Scale .................................................................................................................. 37

Table 3.7 Construct Validity Evidence for Openness to Feedback and


Coaching Scale ........................................................................................................................ 38

VIII

TABLES & FIGURES

Table 3.8 Construct Validity Evidence for Verbal Information Processing


Scale ............................................................................................................................................ 39

Table 3.9 Construct Validity Evidence for Numerical Information Processing


Scale ............................................................................................................................................ 39

Table 3.10 Correlations with Threat Avoidance vs. Reward Seeking and
Observer Ratings .................................................................................................................... 41

Table 3.11 Correlations with Tactical vs. Strategic Thinking and Observer
Ratings ....................................................................................................................................... 42

Table 3.12 Correlations with Data-Driven vs. Intuitive Decisions and


Observer Ratings .................................................................................................................... 42

Table 3.13 Correlations with Defensive vs. Cool-Headed Reactions and


Observer Ratings .................................................................................................................... 43

Table 3.14 Correlations with Denial vs. Acceptance and Observer Ratings ... 43

Table 3.15 Correlations with Superficial vs. Genuine Engagement and


Observer Ratings .................................................................................................................... 44

Table 3.16 Correlations with Openness to Feedback & Coaching and


Observer Ratings .................................................................................................................... 44

Table 3.17 Correlations with Verbal Information Processing and Observer


Ratings ....................................................................................................................................... 45

Table 3.18 Correlations with Numerical Information Processing and Observer


Ratings ....................................................................................................................................... 45

Figure 4.1 Information Processing Styles .................................................................... 49

Figure 4.2 Decision-Making Styles ................................................................................. 57

Figure 5.1 Participant Login Web Page ........................................................................ 66

IX

TABLES & FIGURES

Figure 5.2 Participant Information Web Page ............................................................ 67

Figure 5.3 Participant Menu Wed Page ........................................................................ 68

Figure 5.4 Numerical Section Instructions .................................................................. 69

Figure 5.5 Numerical Section Sample Item ................................................................ 70

Figure 5.6 Numerical Section Sample Page ................................................................ 71

Figure 5.7 Numerical Section Review Page ................................................................ 72

Figure 5.8 Out of Time Page ............................................................................................. 73

Figure 5.9 Verbal Section Instructions ......................................................................... 74

Figure 5.10 Verbal Section Sample Item ..................................................................... 75

Figure 5.11 Verbal Section Sample Page .................................................................... 76

Figure 5.12 Verbal Section Review Page ..................................................................... 77

Figure 5.13 Decision-Making Style Section Instructions ....................................... 78

Figure 5.14 Decision-Making Style Section Sample Item ...................................... 79

Figure 5.15 Decision-Making Style Section Sample Page ..................................... 81

Figure 5.16 Decision-Making Style Section Review Page ...................................... 82

Table B.1 Correlations Between Judgment Scales and HPI Scales &
Subscales ............................................................................................................................... 107

Table B.2 Correlations Between Judgment Scales and HDS Scales &
Subscales ............................................................................................................................... 108

Table B.3 Correlations Between Judgment Scales and CPI Scales ................. 109

X

TABLES & FIGURES

Table B.4 Correlations Between Judgment Scales and NEO PI-R Scales &
Facets ...................................................................................................................................... 110

Table B.5 Correlations Between Judgment Scales and IPIP Big 5 20-Item
Scales ...................................................................................................................................... 110

Table B.6 Correlations Between Judgment Scales and 16PF Scales ............. 111

Table B.7 Correlations Between Judgment Scales and TCI Scales ................. 111

Table B.8 Correlations Between Judgment Scales and MPQ Scales ............... 112

Table B.9 Correlations Between Judgment Scales and 6FPQ Factors & Facets
.................................................................................................................................................... 112

Table B.10 Correlations Between Judgment Scales and PRS Scales ............. 113

Table B.11 Correlations Between Judgment Scales and MVPI Scales &
Themes .................................................................................................................................... 114

Table B.12 Correlations Between Judgment Scales and CISS Interest & Skill
Scales ...................................................................................................................................... 115

Table B.13 Correlations Between Judgment Scales and JPI-R Scales ........... 116

Table B.14 Correlations Between Judgment Scales and HBRI Scales ........... 116

Table B.15 Correlations Between Judgment Scales and SPA Cognitive


Abilities ................................................................................................................................... 116

Table B.16 Correlations Between Judgment Scales and BRI Cognitive Content
.................................................................................................................................................... 117

Table B.17 Judgment Scale Correlations with Observer Adjective Ratings . 118

Table B.18 Judgment Scale Correlations with Observer Ratings on


Descriptive Phrases ............................................................................................................ 119

XI

1. CONCEPTUAL BACKGROUND

1. Conceptual Background

1.1 Introduction

Queen Elizabeth Is move to fight the Spanish Armada, Lincolns emancipation of the slaves,
and Gandhis choice of nonviolent revolution are historical examples of decisions with
positive outcomes (Axelrod, 2006). Conversely, Hannibals invasion of Rome, the Trojans
acceptance of a gift horse, and General Custers tactical moves at Little Bighorn are
decisions that led to disastrous consequences (Weir, 2005).

The business world also provides many examples. Executives at Apple, Johnson & Johnson,
3M, and Intel made decisions that transformed their brands and boosted performance
(Harnish, 2012), but decisions by leaders at Blackberry, Motorola, Lehman Brothers, Kmart,
and Eastman Kodak cost their companies billions of dollars (McIntyre, 2012). Although the
specific content and context of decision-making differ, the consequences of good or bad
judgment are consistent.

Considering the importance of good judgment, Hogan Assessment Systems developed the
Hogan Judgment Report to help organizations understand the decision-making styles of their
key employees. This manual describes: (a) the dynamics of human judgment; (b) the
importance of judgment and decision-making for leader performance; (c) a multi-component
model of judgment, (d) reliability and validity evidence for scales measuring components of
the model; and (e) interpretation, uses, and administration of the Hogan Judgment Report.

The research conducted to develop and validate the Hogan Judgment Model conforms to
standards outlined in the Uniform Guidelines on Employee Selection Procedures (Equal
Employment Opportunity Commission [EEOC], 1978; hereafter Uniform Guidelines), the
Principles for the Validation and Use of Personnel Selection Procedures (Society for
Industrial and Organizational Psychology [SIOP], 2003; hereafter Principles), and the
Standards for Educational and Psychological Testing (American Educational Research
Association [AERA], American Psychological Association [APA], & National Council on
Measurement in Education [NCME], 1999; hereafter Standards). In areas where the
Uniform Guidelines, Principles, or Standards proved vague or inapplicable, we relied on the
broader scientific and professional literature for guidance.

1.2 Heuristics, Biases, and Rationalization

People use a variety of methods to make decisions: logical reasoning; reviewing available
data and background information; or applying heuristics (Gigerenzer & Gaissmaier, 2011).
Logic- and data-based judgments are cognitively demanding, requiring knowledge of
alternatives and their consequences and probabilities (Simon, 1979). In contrast, heuristics
are mental shortcuts that require less cognitive effort (Shah & Oppenheimer, 2008).

Historically, researchers have linked using logic and data with accurate judgment and using
heuristics with error-prone judgment (Kahneman 2003; Tversky & Kahneman 1974).
However, increasing awareness of the complexity of real-world decision-making contexts
suggests that rational and data-driven reasoning are often impractical and may not
guarantee effective outcomes (Busenitz & Barney, 1997; Gigerenzer & Gaissmaier, 2011).
1

1. CONCEPTUAL BACKGROUND

According to Gigerenzer and Gaissmaier (2011), a heuristic is a strategy that ignores part of
the information, with the goal of making decisions more quickly, frugally, and/or accurately
than more complex methods (p. 454). Thus, people often balance speed against decision
accuracy (Wilson, 2010). Heuristics are most useful when the cost of cognitive effort
outweighs the benefits of accuracy (Payne, et al., 1993; Pachur & Hertwig, 2006; Shah &
Oppenheimer, 2008). Examples of common heuristics include:

Affect The affect heuristic is an emotion-based shortcut that relies on positive or


negative feelings attached to people, objects, or situations (Finucane, Alhakami, Slovic,
& Johnson, 2000; Spence & Townsend, 2008; Zajonc, 1980). For example, if an
individual dislikes strangers, he/she may decide to avoid them regularly.

Availability The availability heuristic involves estimating the likelihood of an event


happening based on how easily one can recall similar events (Kliger & Kudryavtsev,
2010; Tversky & Kahneman, 1974; Wilson, 2010). For example, people may assess the
likelihood of having a heart attack by recalling the number of family members who have
had heart attacks.

Effort The effort heuristic is often used to set the value of something based on the
effort expended in trying to obtain it (Kruger, Wirtz, Van Boven, & Altermatt, 2003). For
example, students may believe they deserve higher grades if they put more effort into
class projects.

Researchers speculate about the mechanisms people use to select heuristics. Rieskamp
and Otto (2006) propose that people select heuristics based on past learning experience.
Others (e.g., Snook, et al., 2004) suggest explicit teaching and imitation affect the selection
of heuristics. We think people select heuristics based on their personalities. Regardless of
the underlying mechanisms, however, personal preferences for certain heuristics contribute
to individual differences in judgment and decision-making.

Biases represent another major cognitive shortcut (Busenitz & Barney, 1997). Aronson,
Wilson, and Akert (2010) suggest people distort information to fit preconceived notions
and/or align current situations with previous experiences. As with heuristics, biases
consume few cognitive resources. Examples of common biases include:

Anchoring and Adjustment The anchoring and adjustment bias involves making an
estimate based on an initial anchor value (provided or self-generated) and then
adjusting up or down until a final answer is reached (Tversky & Kahneman, 1974). This
bias is common in sales, where a salesman may provide an unreasonably high price for
an item and then adjust the price to something that seems more reasonable in
comparison.

Gamblers Fallacy The gamblers fallacy involves believing that chance is a self-
correcting process, so that an outcome in one direction will increase the likelihood of an
outcome in the opposite direction (Tversky & Kahneman, 1974). For example, a person
may believe a coin that has landed on heads five times in a row is likely to land on
tails next because it is due.
2

1. CONCEPTUAL BACKGROUND

Escalation of Commitment Escalation of commitment involves continuing a course of


action because of previous investments (Schmidt & Calantone, 2002). This explains why
people often spend more at auctions than they intend. Once they make a bid, they
mentally commit money to it. Committing more money in small increments may seem
easier than backing away from the bidding process.

Framing The framing bias occurs when small changes are made in the way a problem
is presented, leading a decision maker to pick a positively framed option over a
negatively framed one (Hodgkinson, Maule, Bown, Pearman, & Glaister, 2002). For
example, people are willing to spend more for ground beef when it is labeled 80% lean
than 20% fat (Levin, et al., 2002).
Although heuristics and biases are adaptive in some circumstances, they can lead to poor
decisions. One common reaction to bad decisions is rationalization. According to Aronson,
Wilson, and Akert (2010), recognizing bad decisions challenges our positive self-perceptions
and creates cognitive dissonance. We may try to reduce this dissonance by (a) changing our
behavior, (b) changing our beliefs, or (c) adding new beliefs. In other words, we may try to
correct our mistakes or we may rationalize our poor decision. As with heuristics and biases,
there are individual differences in rationalization across people.

Overall, the study of human judgment indicates that individuals (a) often make irrational
decisions, relying on heuristics and biases; (b) show predictable individual differences in
their decision-making tendencies; and (c) react differently to the cognitive dissonance
created by feedback regarding poor decisions. Understanding individual differences in how
individuals approach decisions and react to feedback about poor decisions is critical for
leaders whose decisions impact their organizations.

1.3 Judgment and Leadership

Leaders mostly make decisions about strategy, staffing, and capital investments. Thus,
good judgment is an essential component of effective leadership at both macro and micro
levels.

At the macro level, good judgment drives organizational profitability. Using a decision
experiment, De Neys, et al. (2011) concluded that people who are willing to accept uneven
splits of gains with their partners achieve higher overall payoffs. A study of more than 1,000
major business investments showed that when organizations work to reduce biases in their
decision-making processes, they achieve higher returns (Lovallo & Sibony, 2010).

At the micro level, the leaders decisions affect not only organizational performance, but also
everyone associated with them (Safi & Burrell, 2007). Consider the decisions made
regarding Hurricane Sandy, the coaching scandal at Penn State University, or the use of
chemical weapons in Syria. In each case, the decisions had significant consequences in
some cases life and death for larger populations. Similarly, Russ, et al. (1996) found that
the decision-making style of sales managers impacted employee performance, satisfaction,
and commitment.

3

1. CONCEPTUAL BACKGROUND

To understand decision-making, Hogan developed a model that defines individual


differences in good judgment in terms of information processing style, tendencies in
approaching decisions, and reactions to feedback about bad decisions. This assessment
model facilitates leadership development by building strategic self-awareness regarding how
leaders process information, approach business decisions, and react to feedback regarding
decision outcomes.

1.4 Current State of Judgment and Decision-Making Research

Researchers have focused on classifying human judgment processes. In the emerging field
of decision science, researchers combine Economics (e.g., Rubinstein, 1998), Cognitive
Psychology (e.g., Tversky & Kahneman, 1974), and Neuroscience (e.g., Bechara, 2005;
Brocas, 2012) to understand decision-making. However, in reality people are seldom able
to follow step-by-step logic processes (Mintzberg & Westley, 2001). For example, rather
than thinking first (i.e., making decisions based on facts), people may see first (i.e.,
make decisions based on insights) and do first (i.e., making decisions through
experimentation). After observing mid-career managers from across the globe, Mintzbeg
and Westley (2001) conclude that thinking first is only functional when the issues are
clearly defined, the context is structured, and reliable data are available, whereas seeing
first and doing first enable visioning and venturing under novel and ambiguous situations.

Although rationalization may sometimes lead to more accurate decisions, real decision-
makers always make decisions under time pressure in unstructured contexts. Thus,
knowing the strengths and challenges of a leaders decision-making style provides insights
into leadership competence.

In addition to ignoring individual differences in irrational decision-making tactics, previous


research also overlooks individual differences in reactions to decision outcomes. Becoming
aware of such reactions is critical for leadership performance. The complexity of global
business increases the difficulty of decision-making, which necessarily increases the
incidence of biased and/or ineffective decisions. For example, a study of 356 decisions
made in medium to large organizations in the U.S. and Canada reveals that 50% of
managers decisions fail (Nutt, 1999; 2004). Given the consequences of leader decisions
and the high rate of wrong decisions, it is critical to explore ways to improve leader judgment
and decision-making.

Understanding decision-making preferences is necessary but insufficient for improving


judgment. Kahneman et al. (2011) conclude that becoming aware of judgment biases only
minimally improves decision quality. Similarly, Milkman, et al. (2009) suggest going beyond
the description of judgment biases to improve decision-making. One way to do this is to
understand how people react to feedback about incorrect decisions. According to Fischhoff
(1982), the best way to overcome judgment bias is to combine feedback with coaching.
Brett and Atwater (2001) find people with a learning goal orientation are particularly
receptive to negative feedback, leading to a greater likelihood of performance improvement.
Similarly, Smither, et al. (2005) find leaders who react positively to feedback are more likely
to benefit from multi-source feedback. These findings suggest there are important
individual difference in receptiveness to feedback and greater receptivity is associated with
greater benefits.
4

1. CONCEPTUAL BACKGROUND

Feedback is particularly important for leaders, because as job level increases, they are
responsible for making higher stakes decisions, but are also less likely to receive
constructive feedback (Gallo, 2012). Moreover, leaders who are uncomfortable offering
criticism are also less interested than others in receiving feedback (Kouzes & Posner, 2014;
Zenger & Folkman, 2014). The Dunning-Kruger effect (Kruger & Dunning, 1999) suggests
that poor performers not only make bad decisions, but also overestimate their talent. This
can lead to a vicious cycle, especially for leaders whose decisions are critical to
organizational performance.

1.5 The Hogan Judgment Model

To fill these gaps in existing research on judgment, we developed the Hogan Judgment
Model presented in Figure 1.1. This model: (a) accounts for individual differences in how
leaders process information, (b) defines individual differences in pre-decision tendencies,
and (c) examines post-decision reactions to feedback. Thus, this model provides a
comprehensive approach to assessing business judgment by focusing on all the factors
consequential to leadership judgment in the global business environment. Below, we
introduce each component of the model in detail.

Figure 1.1
Hogan Judgment Model

1.5.1 Information Processing

Information processing style concerns how individuals acquire and process information.
Information processing shapes the content of business decisions (Citroen, 2011). Although
leaders typically have adequate cognitive ability, they differ in terms of their information
preferences. The Hogan Judgment Model focuses on two types of information: (a) numerical
data, and (b) verbal information. Vocational interest research shows people can be sorted
into two broad groups: scientists and humanists (Ackerman & Heggestad, 1997; Snow,
1956). Scientists enjoy analyzing and interpreting objective problems rooted in data,

5

1. CONCEPTUAL BACKGROUND

whereas humanists enjoy creating, analyzing, and interpreting emotional experiences. The
information processing styles of these two tribes are distinct. Scientists are more effective
at processing numerical data and identifying patterns and rules, whereas humanists are
more effective at processing verbal information and understanding and using words to
create desired effects.

Decision-makers tend to specialize in either numerical data or verbal information. Although


people can process both types of information, many process one type of information more
efficiently than the other.

Understanding the information processing styles of decision-makers can help organizations


diversify their leadership teams and ensure that leaders have the right sources of
information to make effective decisions. Similarly, individual leaders can use this
information to develop their judgment by playing to their natural strengths and developing
attributes that dont come as naturally.

1.5.2 Using Personality to Assess Decision-Making Approaches

Personality predicts a range of organizational performance measures, including overall job


performance (Barrick, Stewart, & Piotrowski, 2002; Dudley, Orvis, Lebiecki, & Cortina, 2006;
J. Hogan & Holland, 2003), team performance (Peeters, Van Tuijl, Rutte, & Reymen, 2006),
expatriate performance (Mol, Born, Willemsen, & Van Der Molen, 2005), Counterproductive
Work Behaviors (CWB; Berry, Ones, & Sackett, 2007; Gruys & Sackett, 2003), Organizational
Citizenship Behaviors (OCBs), altruism, job dedication, interpersonal facilitation, and
generalized compliance (Borman, Penner, Allen, & Motowidlo, 2001; Dubley et al., 2006;
Hurtz & Donovan, 2000; LePine, Erez, & Johnson, 2002; Organ & Ryan, 1995). Personality
also predicts individual differences in judgment and decision-making.

Soane et al. (2010) show that personality predicts risky choices in a wide variety of social
situations. Similarly, higher impulsivity is associated with faulty decision-making (Davis,
Patte, Tweed, & Curtis, 2007). Dewberry et al. (2013) conclude that Five-Factor Model (FFM)
personality variables explain variance in decision-making competence beyond decision-
making and cognitive styles. In particular, Extraversion is negatively associated with
competent decision-making, whereas Emotional Stability is positively associated with
decision-making. Examining the dark side of personality, Zhu and Chen (2014) suggest that
narcissism is related to CEOs tendency to adopt the corporate strategies of other boards.
Chen et al. (2014) also find that CEO overconfidence influences responses to corrective
feedback (e.g., inputs from markets, customers, employees, and other stakeholders).
Similarly, Tang et al. (2014) propose that CEO hubris negatively impacts corporate social
responsibility. Consistent with this research, the Hogan Judgment Report uses personality
measures to assess individual differences in decision-making.

Research on individual differences in judgment and decision-making has typically focused


on judgment biases and their impact on real-life outcomes (e.g., Bruine de Bruin, et al.,
2007; Stanovich & West, 2000). Measures of judgment (e.g., Bruine de Bruin et al., 2007;
Bruine de Bruin et al., 2012; Parker & Fischhoff, 2005) study generic biases that may not
impact business decisions. In contrast, the Hogan Judgment Model concerns three

6

1. CONCEPTUAL BACKGROUND

dimensions that relate to business decisions: (a) threat avoidance vs. reward seeking; (b)
tactical vs. strategic thinking; and (c) data-driven vs. intuitive decisions.

1.5.2.1 Threat Avoidance vs. Reward Seeking

All decisions entail potential threats and rewards. For example, when Marissa Mayer was
appointed CEO of Yahoo and banned staff telecommuting, she thought that enhanced
employee collaboration would outweigh the risk of degrading staff morale.

Individuals differ in their tendencies to avoid threats or seek rewards. According to loss-
aversion theory, people are generally prone to negativity bias (e.g., Baumeister et al., 2001;
Rozin & Royzman, 2001), which means they pay more attention to potential threats than
rewards when evaluating decision options (e.g., Benartzi & Thaler, 1995; Kliger & Levit,
2009; McGraw et al., 2010). People also vary in their risk perceptions. For example,
entrepreneurs perceive risks as less threatening (Cooper, Woo, & Dunkelberger, 1988) and
focus on potential future rewards.

McGhee, et al. (2012) found that Extraversion, Openness, and (low) Conscientiousness
predict risk-taking in preadolescents. Similarly, research with adults suggests that
Extraversion, positive affectivity, and sensitivity to Behavior Activation Systems (BAS) are
associated with reward seeking, whereas Neuroticism, negative affectivity, and sensitivity to
Behavior Inhibition Systems (BIS) are associated with failure avoidance (Atkinson, 1957;
Elliot & Thrash, 2002). Personality variables such as sensation seeking, impulsivity, and low
self-control are also associated with risky behavioral choices (Zuckerman, 2007).

Organizations often need decision-makers with different levels of threat avoidance or reward
seeking. For example, start-up companies need employees who will make bold decisions to
expand the companys bottom line, whereas investment advisors should make decisions
that minimize potential threats to their clients finances (Weber, et al., 2002). Individuals
can also use information about their tendency to avoid threats or seek rewards to develop
their decision-making skills.

1.5.2.2 Tactical vs. Strategic Thinking

In approaching decisions, some people focus on immediate, practical issues, whereas


others focus on longer term challenges and opportunities. Tactical thinkers tend to evaluate
cost, implementation, and staffing problems to the exclusion of longer term contextual
issues (Leonard & McAdam, 2002). Conversely, strategic thinkers tend to consider big
picture trends, capabilities, and sustainable gains (Citroen, 2011), but may neglect
important logistical matters. Sustained success and growth requires both tactical execution
and strategic vision. For example, Apples success reflects both Steve Jobs vision and the
tactical execution of his designs.

According to Pellegrino (1996), certain personality characteristics predict strategic thinking.


Kauer, et al. (2007) found that, in top management teams, flexibility, achievement
motivation, networking skill, and action orientation impact agenda-setting, generating
strategic alternatives, and speed of strategic decision-making. Similarly, OConnell et al.

7

1. CONCEPTUAL BACKGROUND

(2013) found that personality predicts preferences for long-term vs. short-term rewards
when people make decisions for others (O'Connell, et al., 2013).

Organizations can use information about the tactical and strategic thinking preferences of
their employees to select, position, and develop key talent. Strategic thinkers are critical to
large-scale direction setting, whereas tactical thinkers are needed to implement this
direction and maintain established structures (Leonard & McAdam, 2002). Similarly,
individuals can use strategic self-awareness of their tendencies towards tactical or strategic
thinking to ensure they consider both short- and long-term goals.

1.5.2.3 Data-Driven vs. Intuitive Decisions

Some people prefer to use data and facts to make decisions, whereas others prefer using
intuition and experience as their guide. According to Kahneman (2003), peoples decisions
are affected by two interacting systems: System 1 (intuition) and System 2 (deliberative
thinking). Intuitive decision-makers prefer System 1, which is fast, automatic, and effortless.
McNaught (2013) found that experienced leaders often rely on intuition. In contrast, data-
driven decision makers prefer System 2, which is slow, controlled, and effortful. People who
prefer System 2 collect as much information as possible regarding relevant alternatives,
potential consequences, and probabilities (Simon, 1979).

Deliberate thinking is positively correlated with a need for structure, need for cognition,
Conscientiousness, and perfectionism, whereas intuitive thinking is correlated with
Extraversion, Agreeableness, and Openness to Experience (Betsch, 2008)i.e., both are
related to personality. These differences in style are not necessarily correlated with decision
accuracy. Data-driven decisions may seem more accurate because they are based on
evidence, but the complexity of the global business environment often limits the amount of
information available to support optimal decision-making. Moreover, information may
quickly become outdated, which challenges the accuracy of data-driven decisions
(Gigerenzer & Gaissmaier, 2011). In contrast, experienced leaders believe the high quality
of intuitive decisions is a function of their experience (McNaught, 2013), although the
accuracy of such decisions also seems limited (Choudhry, Fletcher, & Soumerai, 2005;
Swets, Dawes, & Monahan, 2000; Tetlock, 2005).

Whether data-driven or intuitive decision-making is more accurate depends on many factors.


Within any organization, some decisions require careful consideration, whereas others must
be made quickly. Individuals can use information concerning their own style to understand
when they are taking too long or jumping too quickly into decisions.

1.5.3 Using Personality to Assess Reactions to Feedback

After people make decisions, personality predicts their reactions to feedback about their
decisions. Although past research demonstrates that performance can be improved with
feedback (Ilgen, Fisher, & Taylor, 1979; Kluger & DeNisi, 1996; Latham & Locke, 2007;
Salmoni, Schmidt, & Walter, 1984), meta-analytic research shows that feedback often
reduces performance (Kluger & DeNisi, 1998), suggesting that there are important
individual differences in reactions to feedback. A recent Harvard Business Review article
(Zenger & Folkman, 2014) shows that among 2,500 people surveyed about reactions to
8

1. CONCEPTUAL BACKGROUND

feedback, 52.0% found negative feedback helpful, whereas 47.5% preferred to receive
positive feedback. Personality characteristics such as self-esteem and resiliency predict
their acceptance of feedback (How Employees Process, 2013; Shrauger & Rosenberg,
1970). For example, individuals with low self-esteem perceive critical feedback as too harsh,
which inhibits their task performance. In addition, Steers (1975) found that, for high
achievement individuals, there is a positive relationship between the amount of feedback
provided and subsequent performance. Consistent with this, the Hogan Judgment Report
assesses individual differences in reactions to feedback about ineffective decisions.

Feedback is critical for any successful developmental initiative, as shown by research on


goal-setting and self-regulatory motivation (Button, Mathieu, & Zajac, 1996; Donovan &
Williams, 2003; Locke & Latham, 1990). Individuals who react positively to feedback are
more likely to set difficult and specific goals (Ilies & Judge, 2005; Ilies, Judge, & Wagner,
2010), which leads to increases in individual and organizational performance (e.g., Latham
& Locke, 2007; Locke, 1997; Locke & Latham, 1990).

Personality predicts how people respond to negative feedback. For example, Bell and Arthur
(2008) found that Extraversion and Agreeableness positively affect feedback acceptance.
Similarly, Smither, et al. (2005) found that leaders higher in Adjustment, Sociability, and
Conscientiousness were more willing to accept feedback, seek additional feedback, and use
feedback to improve performance. The Hogan Judgment Model characterizes reactions to
feedback about ineffective decisions along three dimensions: (a) defensive vs. cool-headed;
(b) denial vs. acceptance; and (c) superficial vs. genuine engagement.

1.5.3.1 Defensive vs. Cool-Headed

Some individuals respond to negative feedback by becoming emotionally volatile and


defensive, and blaming external factors beyond their control. Such individuals tend to be
challenged in their ability to recognize how they have contributed to the mistakes and how
they can improve their performance in the future. In contrast, individuals who remain calm
after negative feedback are more likely to recognize personal faults and set goals for
improvement. For example, individuals with an internal locus of control are more likely to
accept feedback than individuals with an external locus of control because they take
ownership of their behaviors and are motivated to make changes (Feather, 1968).

1.5.3.2 Denial vs. Acceptance

According to Freud (1946), denial is a common defense people use to protect themselves
from negative thoughts and feelings. Some people react to negative feedback by denying
the data, whereas others more readily accept the criticism. Although denial can make
people feel better, it does not help them make better decisions in the future. For example,
Kluger and DeNisi (1998) found that individuals who take negative feedback personally are
unlikely to improve their subsequent performance.

9

1. CONCEPTUAL BACKGROUND

1.5.3.3 Superficial vs. Genuine Engagement

Some peoples reactions to feedback are motivated by a desire to maintain positive social
relations (Aitkenhead, 1984). According to Ashford and Northcraft (1992), the tendency to
manage impressions often overrides listening to feedback in evaluative contexts. Research
on collegiate swimmers found that those with high self-esteem find negative feedback
motivating (Marsden, 1998). Similarly, business leaders who genuinely listen to negative
feedback are more likely to make better decisions in the future than leaders who listen to
feedback only to maintain positive social impressions.

In general, the more calm, accepting, and engaged a person is when receiving negative
feedback, the more coachable that person is, and the more likely they are to improve future
decisions. That is, individuals who are receptive to coaching are more likely to recognize
their mistakes, take responsibility for them, learn from them, and improve future decisions
compared to individuals who are generally resistant to coaching. Furthermore, although all
three characteristics relate to reactions to feedback, they are independent. To improve
decision-making, one must consider his/her tendencies on all three dimensions and reflect
on these tendencies when faced with negative feedback.

10

2. INVENTORY CONSTRUCTION, RELIABILITY, & STRUCTURAL PSYCHOMETRICS

2. Inventory Construction, Reliability, and Structural Psychometrics

2.1 Development

As noted in Chapter 1, the Hogan Judgment Model assumes that, although individual
judgment and decision-making is biased and irrational, it is still predictable (Ariely, 2008).
Although some cognitive shortcuts (i.e., heuristics; Gigerenzer & Gaissmaier, 2011; Shah &
Oppenheimer, 2008) are advantageous, others (i.e., biases; Busenitz & Barney, 1997)
cause people to distort information and make bad decisions. Moreover, individuals
typically rationalize their biased decision-making after the fact; as a result, they internalize
accurate decisions and blame failed decisions on external sources (Aronson, Wilson, &
Akert, 2010). How leaders make decisions and respond to failure is especially
consequential because they control resources (Lovallo & Sibony, 2010) and typically make
bad decisions at least half of the time (Nutt, 1999; 2004).

Researchers usually attempt to assess good judgment as the ability to make correct
decisions. These efforts often focus on critical thinking, IQ, and other cognitive measures.
However, this approach (a) ignores non-cognitive individual differences that affect decision-
making, (b) provides no information for developing judgment, and (c) ignores the fact that
highly intelligent people often make bad decisions. To understand this last issue, some
researchers have studied how personality and values affect decision-making. However, few
have studied the interaction between cognitive and non-cognitive attributes.

In our view, people with good judgment tend to not only make good decisions, but also
recognize their mistakes and take corrective action. Our assessment concerns individual
differences in (a) information processing, (b) decision-making, and (c) reactions to feedback
about failed decisions.

2.1.1 Information Processing

For the cognitive component of the judgment assessment, we drew from research on
learning potential and vocational interests.

Learning potential concerns the ability to acquire knowledge and information (Hamers &
Ruijssenaars, 1992). Some people learn more quickly than others. Faster learners have
advantages in educational settings, and in complex, rapidly-changing work environments.
Also, the more complex a job, the less incumbents can rely on previously acquired
knowledge to solve problems. Thus, learning potential is consequential in complex jobs.

Research on vocational interests shows that people can be sorted into two tribes: scientists
and humanists (Snow, 1956; Ackerman & Heggestad, 1997). Scientists enjoy analyzing and
interpreting data, whereas humanists enjoy creating, analyzing, and interpreting powerful
emotional experiences. Science is about prediction and control, and the humanities give
form to feeling. These tribes are also characterized by distinct numerical and verbal learning
styles. Numerical learning potential concerns the ability to identify patterns and rules, which
predicts interest in science and math and performance in fields such as finance, accounting,
engineering, and IT. Verbal learning potential concerns the ability to understand and use

11

2. INVENTORY CONSTRUCTION, RELIABILITY, & STRUCTURAL PSYCHOMETRICS

words to create desired effects, which predicts interest in literature, philosophy, journalism,
and advertising and performance in fields requiring good communicational skills. Numerical
and verbal potential are related, such that some individuals are slow and deliberate in
processing both types of information and others are able to quickly and efficiently process
both types. However, many individuals tend to prefer one type of information over the other.

Based on these considerations, we included numerical and verbal elements in the cognitive
component of the Hogan Judgment Model. We chose these not only because of their
importance in the vocational interest literature, but also because they represent the sources
of information most frequently used in making organizational decisions. That is, before
making a decision, leaders typically refer to numerical data (e.g., sales, stock performance)
and/or verbal information (e.g., emails, reports) to inform their decisions. We wrote two sets
of parallel items for each section.

2.1.1.1 Numerical Information Processing

For numerical information processing, we wrote items using multiple formats including
balancing equations, identifying larger or smaller fractions, sequence completion, and
pattern identification and completion. Although all items could be answered correctly by
most professionals with unlimited time and resources, we imposed a 10-minute limit. This
reflects the fact that real-world decision-makers have limited time to review data before
making a decision.

We began testing these items in the fall of 2011 using the Numerical Reasoning Test 20
(NRT-20; Chamorro-Premuzic, 2008) as a guide. We chose the NRT-20 as a marker
because it does not require previous knowledge or specialized training in mathematics. We
tested new items with research samples and revised items as needed to improve their
psychometric properties. We also solicited input from experts and colleagues concerning
item content and accuracy.

We collected data through Hogans on-line platform. The system is fully redundant, using a
multi-location systems architecture to ensure constant accuracy and availability.

2.1.1.2 Verbal Information Processing

For verbal information processing, we wrote items requiring users to solve problems
concerning the placement of letters in the alphabet, types of items, analogies, opposites,
object size, and meanings. We wrote half using positive wording (e.g., X is a type of Y) and
the other half using negative wording (e.g., X is not a type of Y). The items require test-
takers to recognize patterns in verbal data and use that information to determine whether
the item is True or False. We also imposed a two-minute time limit. This limit helps us
better determine how quickly individuals can process verbal information, reflecting the fact
that organizational decision-makers have limited time to make decisions.

We began testing these items in the fall of 2012 using assessments designed to measure
working memory (e.g., Baddeley, 1986; 1997) as a starting point. Next, using item types
from other verbal assessments, we began testing items using convenience samples. Again,

12

2. INVENTORY CONSTRUCTION, RELIABILITY, & STRUCTURAL PSYCHOMETRICS

we revised items as needed and solicited feedback from experts and colleagues. We
collected data using the Hogan platform previously described.

The Hogan Judgment Model describes information processing style using one of four
categories: (a) deliberate, (b) qualitative, (c) quantitative, or (d) versatile. We define these
categories in a later section of this chapter.

2.1.2 Decision-Making Approaches & Decision-Making Styles

To assess decision-making approaches, we identified three independent dimensions of


individual preferences in approaching decisions: (a) goal-orientation (i.e., threat avoidance
vs. reward seeking), (b) thinking style (i.e., tactical vs. strategic), and (c) base of information
(i.e., data-driven vs. intuitive).

Next, we identified scales from these assessments that predicted behaviors associated with
our overall three themes (e.g., MVPI Security and threat avoidance; HPI Ambition and
strategic thinking; MVPI Scientific and data-driven decisions) and identified items from the
scales that seemed to reflect each theme (e.g., I dont like unpredictable situations, I am
a leader in my group, I am interested in science, etc.). We also wrote items with work-
related content for each construct. Specifically, we wrote items to reflect what a person with
a given tendency is likely to do when approaching a work-related decision. We developed
these scales with non-overlapping content and homogenous themes to avoid overlap.

We began studying tactical and strategic thinking preferences in 2009, and began writing
items for the remaining pre-decision scales in the fall of 2011. We developed each scale by
combining existing assessment content with new items, testing the scale with research
samples, analyzing the data to calculate statistical properties, and making revisions as
needed to enhance the psychometric properties of each scale. We again solicited expert
feedback and collected data using Hogans platform.

We used the three scales previously described to define eight types, each represented by an
occupation representing a combination of approaches to making business decisions. These
occupational types are not indicative of likely vocational interests or occupational
performance, but are intended to describe an individuals general decision-making style.
For example, a person characterized as reward-seeking, strategic, and data-driven would be
classified as an Investor (e.g., Warren Buffett). Such individuals prefer to maximize their
long-term rewards based on data-driven decisions.

2.1.3 Reactions to Feedback & Openness to Feedback and Coaching

Our strategy for writing items to assess reactions to negative feedback paralleled the
process for developing pre-decision scales, but focused on tendencies individuals use when
reacting to feedback about failed decisions. Again, we started with three general themes:
(a) initial response (i.e., defensive vs. cool-headed reactions), (b) feedback synthesis (i.e.,
denial vs. acceptance), and (c) ongoing engagement (i.e., superficial vs. genuine). The high
end of each scale reflects more positive reactions to feedback, whereas the low end reflects

13

2. INVENTORY CONSTRUCTION, RELIABILITY, & STRUCTURAL PSYCHOMETRICS

tendencies to shoot the messenger, ignore the messenger, and mistrust the
messenger, respectively.

Next, we identified specific scales predictive of outcomes related to our three themes (e.g.,
HDS Excitable and defensive; HDS Bold and denial; HDS Dutiful and superficial
engagement) and items from these scales that directly aligned with each (e.g., I can get
angry quickly, I do many things better than almost everyone I know, I usually try to tell
people what they want to hear, respectively). Again, we wrote items designed to reflect
what a person with a given tendency is likely to do when confronted with negative feedback
about a failed decision. We piloted items and refined scales to improve psychometric
qualities and avoid excessive overlap in content. We began writing items in the fall of 2011.
We developed each scale by combining existing assessment content with new items, testing
the scale with research samples, analyzing the data to calculate statistical properties, and
making revisions to enhance the psychometric properties of each scale. We also received
valuable input from colleagues across the globe. We tested this content using the same
web platform previously described.

Although we developed the pre- and post-decision scales using the same general process,
they differ in two important ways. First, the pre-decision scales describe tendencies that are
neither desirable nor undesirable, whereas low scores on the post-decision scales are
undesirable. Second, we summarize the results differently. We present the results for pre-
decision biases in terms of types, we present the results for post-decision reactions in terms
of scale scores.

2.2 Definitions of the Scales

We define the scales for the three components of the Judgment Model as follows:

Numerical Information Processing concerns the ability to quickly recognize patterns in


numerical data and use those patterns effectively to make relevant decisions.

Verbal Information Processing concerns the ability to quickly recognize and extract relevant
facts from verbal information and use those facts effectively to make relevant decisions.

Information Processing Styles describe a persons preferences for acquiring and processing
information needed to make decisions:
Deliberate information processors take their time in processing both numerical and
verbal information.
Qualitative individuals are more efficient at processing verbal information than they
are at processing numerical data.
Quantitative individuals are more efficient at processing numerical data than they are
at processing verbal information.
Versatile information processors can quickly and effectively process both numerical
data and verbal information.

Threat Avoidance vs. Reward Seeking concerns the degree to which an individual focuses
on potential risks or possible rewards as a goal orientation when approaching decisions.

14

2. INVENTORY CONSTRUCTION, RELIABILITY, & STRUCTURAL PSYCHOMETRICS

Tactical vs. Strategic Thinking concerns a persons tendency to focus on immediate relevant
details or larger, contextual issues.

Data-Driven vs. Intuitive Decisions concerns the degree to which a person bases his or her
decisions on relevant data or past experience.

Decision-Making Styles are defined by combining scores across the three pre-decision
scales. Doing this results in the following 8 decision-making types:
Auditors avoid threats by making tactical decisions grounded in data.
Surgeons make tactical, intuitive decisions to avoid immediate threats.
Stock Traders seek rewards by making tactical, data-based decisions.
Defense Analysts avoid long-term threats by making strategic, data-based decisions.
Politicians seek long-term rewards by making strategic, intuitive decisions.
Chess Players defend against threats by making strategic, experience-based
decisions.
Promoters seek short-term rewards by making tactical, intuitive decisions.
Investors maximize long-term rewards by making strategic, data-based decisions.

Finally we come to reactions to feedback. There are three primary reactions as follows:

Defensive vs. Cool-Headed concerns the degree to which individuals blame bad results on
external factors or remain cool-headed and calmly reflect on their mistakes.

Denial vs. Acceptance concerns the degree to which individuals respond to negative
feedback by denying reality or deflecting problems versus directly addressing the failure.

Superficial vs. Genuine Engagement concerns the degree to which individuals respond to
negative feedback by putting on an act to gain approval versus actively engaging in
feedback to improve future decisions.

Scores on these three dimensions reflect:

Openness to Feedback and Coaching, or an individuals overall readiness to learn from


mistakes and profit from coaching to improve future decision-making:
Resistant individuals may tend to react to negative feedback about failed decisions
by blaming others, denying responsibility, and pretending to care about feedback
without actually engaging in it.
Neutral individuals often seem moderately receptive to feedback, but may also
struggle with tendencies to react poorly to news about failed decisions.
Receptive individuals tend to analyze their missteps and solicit advice about how to
make better decisions in response to a failed decision.

15

2. INVENTORY CONSTRUCTION, RELIABILITY, & STRUCTURAL PSYCHOMETRICS

2.3 Item Composition of the Scales

Each component of the Hogan Judgment Model is assessed using a distinct set of items.
Below, we describe the item composition of the scales in each section.

2.3.1 Numerical Information Processing

The numerical information processing assessment includes two parallel forms, each with 15
items of increasing difficulty. Participants have 10 minutes to answer as many questions as
possible. Each item includes a stimulus, such as requiring the participant to complete
sequences or identify patterns. Once the participant has examined the stimulus, he or she
must make required computations and select the correct answer from a set of five possible
responses. Participants can use scratch paper but not calculators or other external devices.
Participants receive one point for each correct response and no points for incorrect
responses, so total numerical scores range from 0 to 15. Scores of 0 to 7 indicate average
numerical information processing ability; scores 8 to 15 indicate high ability. There is no
item overlap and participants do not need previous knowledge or advanced education in
mathematics to complete the assessment.

2.3.2 Verbal Information Processing

The verbal information processing assessment includes two parallel forms, each containing
48 simple statements answered True or False. These items concern placement of
letters in the alphabet, analogies, opposites, and item types, sizes, and meanings.
Participants have two minutes to answer as many items as possible. They earn one point for
each correct response, so total verbal scores range from 0 to 48. Scores of 0 to 28 indicate
average verbal information processing ability; scores 29 to 48 indicate high ability.
Readability statistics computed on the 48 items indicated an average sentence length of 7.2
words and an average word length of 3.5 characters. A Flesch-Kincaid reading level analysis
shows that the verbal assessment is written at a first-grade level (1.4) with a reading ease of
97.5%, indicating that participants do not need previous knowledge or higher-order
vocabulary to complete this assessment.

2.3.3 Decision-Making Approaches and Reactions to Feedback

The decision-making style assessment presents items for both the pre-decision approach
and the post-decision reaction scales. This 75 item assessment includes 28 items from the
HPI, 19 from the HDS, 14 from the MVPI, and 14 new items. It includes simple statements
participants answer as True if it describes them or False if it does not. We compute raw
scores as the total number of items endorsed for the high end of each scale, and convert
raw scores to normative scores based on a sample representative of the assessments
intended population. For each scale, normative scores are presented in terms of a ratio
indicating the participants natural underlying tendencies for both sides of the scale (e.g., 74%
Strategic, 26% Tactical).

There is no item overlap across the 75 items, which were screened for content participants
might perceive as offensive or invasive. Specifically, there are no items concerning sexual

16

2. INVENTORY CONSTRUCTION, RELIABILITY, & STRUCTURAL PSYCHOMETRICS

preferences, religious or political beliefs, criminal or otherwise illegal behavior, racial/ethnic


attitudes, or perceptions of disabled individuals.

Readability statistics computed on the 75 items indicated an average sentence length of 8.3
words and an average word length of 4.1 characters. A Flesch-Kincaid reading level analysis
shows that the decision-making style assessment is written at a fourth-grade level (4.3) with
a reading ease of 78.4%.

2.4 Descriptive Statistics and Reliabilities

Table 2.1 presents descriptive statistics for each of the scales in the Hogan Judgment Model,
including the number of items in each scale, mean scale scores, standard deviations,
skewness and kurtosis statistics, internal consistency reliability, average inter-item
correlations, and standard errors of measurement.

Skewness refers to departure from symmetry in a distribution of scores. When a distribution


is normal and symmetrical, skewness values are around zero. Positive skewness values
indicate that most scores fall at the bottom end of a distribution, and negative skewness
values indicate that most scores fall on the top end of a distribution. Skewness values
greater than +1.0 or less than -1.0 generally indicate a significant departure from symmetry.

Kurtosis refers to how peaked or flat a score distribution is relative to the normal distribution.
When scores are normally distributed, kurtosis values are around zero and the score
distribution is said to be mesokurtic. When the distribution is sharper than the normal
distribution, kurtosis values are positive and the score distribution is said to be leptokurtic.
When the distribution is broader than the normal distribution, kurtosis values are negative
and the score distribution is said to be platykurtic. Kurtosis values of more than twice the
standard error indicate a significant departure from the normal distribution.

Internal consistency is a form of reliability estimating how well items from a single scale
estimate a common attribute. When items measure the same construct, internal
consistency reliability is high. When items measure different constructs, it is low. Average
correlations between items also provide information about internal consistency (DeVellis,
1991), where higher average inter-item correlations indicate that items measure the same
construct. Clark and Watson (1995) suggest that average inter-item correlations be at
least .15, although acceptable thresholds may be lower depending on the heterogeneity of
the construct.

We also provide two estimates for standard error of measurement one based on observed
scores (SEM1), and the other (SEM3) based on true score estimates (Dudek, 1979).
Standard error of measurement represents the standard deviation of a persons scores if he
or she were to complete the assessment repeatedly. Lower estimates indicate the results
are more stable.

To examine the descriptive statistics and reliability, we obtained data from a global sample
of executives, managers, and other high-level professionals who completed the instrument
as part of leadership development or research. On average, participants were 54.96 years

17

2. INVENTORY CONSTRUCTION, RELIABILITY, & STRUCTURAL PSYCHOMETRICS

old (SD = 17.03), and the sample included 112 males and 171 females (248 did not
indicate their sex). Table 2.1 presents descriptive statistics, reliability estimates, and
standard errors.

Table 2.1
Classical Scale Statistics for Hogan Judgment Model Scales
Scale Items M SD Skew Kurt Avg. r SEM1 SEM3
Numerical 15 7.32 2.50 .02 .06 .63 .10 1.52 1.94
Verbal ^ 48 20.84 6.20 .28 .00 .94 .26 1.52 2.12
Threat vs. Reward 14 8.01 3.46 -.34 -.63 .80 .22 1.55 2.08
Tactical vs. Strategic 13 8.21 3.42 -.51 -.63 .82 .26 1.45 1.96
Data-Driven vs. Intuitive 12 5.61 3.40 .16 -1.06 .82 .28 1.44 1.95
Defensive vs. Cool-Headed 13 8.82 3.21 -.85 .06 .80 .25 1.44 1.93
Denial vs. Acceptance 11 4.93 2.65 .23 -.66 .73 .20 1.38 1.81
Superficial vs. Genuine 12 7.28 2.69 -.40 -.34 .69 .16 1.50 1.95
Note: N = 502; ^ = Verbal item sub-sample N = 135; Items = Number of items in each scale; M = Scale
mean; SD = Standard deviation; Skew = Skewness statistic; Kurt = Kurtosis statistic; = Cronbachs alpha
internal consistency reliability; Avg. r = Average inter-item correlation; SEM = Standard error of
1

measurement applied to an individuals estimated true score given their observed score; SEM = Standard
3

error of measurement applied to an individuals observed score.

For numerical and verbal information processing, average scores are near scale midpoints,
show adequate variability, and demonstrate no significant departures from symmetry.
Reliability estimates and average inter-item correlations indicate both scales are internally
consistent and conceptually related.

For the pre-decision scales, mean scores range from 5.61 (Data-Driven vs. Intuitive
Decisions) to 8.21 (Tactical vs. Strategic Thinking), with standard deviations between 3.40
(Data-Driven vs. Intuitive Decisions) and 3.46 (Threat Avoidant vs. Reward Seeking).
Skewness statistics indicate that score distributions are adequately symmetrical, ranging
from -.51 (Tactical vs. Strategic Thinking) to .16 (Data-Driven vs. Intuitive Decisions).
Kurtosis statistics also indicate that the pre-decision scales are not abnormally peaked or
flat, ranging from -1.06 (Data-Driven vs. Intuitive Decisions) to -.63 (Threat Avoidant vs.
Reward Seeking; Tactical vs. Strategic Thinking). Cronbachs alpha reliability estimates
illustrate that these scales are internally consistent, ranging from .80 (Threat Avoidant vs.
Reward Seeking) to .82 (Tactical vs. Strategic Thinking; Data-Driven vs. Intuitive Decisions),
with an average reliability of .81 across scales. Average inter-item correlations range
from .22 (Threat Avoidant vs. Reward Seeking) to .28 (Data-Driven vs. Intuitive Decisions),
averaging .25 across scales. As a whole, these results provide support for the psychometric
qualities of the pre-decision scales.

For the post-decision scales, mean scores range from 4.93 (Denial vs. Acceptance) to 8.82
(Defensive vs. Cool-Headed), with standard deviations between 2.65 (Denial vs. Acceptance)
and 3.21 (Defensive vs. Cool-Headed). Skewness statistics indicate that score distributions
are adequately symmetrical, ranging from -.85 (Defensive vs. Cool-Headed) to .23 (Denial vs.
Acceptance). Kurtosis statistics also indicate that the post-decision scales are not
abnormally peaked or flat, ranging from -.66 (Denial vs. Acceptance) to .06 (Defensive vs.
Cool-Headed). Cronbachs alpha reliability estimates illustrate that these scales are
internally consistent, ranging from .69 (Superficial vs. Genuine Engagement) to .80

18

2. INVENTORY CONSTRUCTION, RELIABILITY, & STRUCTURAL PSYCHOMETRICS

(Defensive vs. Cool-Headed), with an average reliability of .74 across scales. Average inter-
item correlations range from .16 (Superficial vs. Genuine Engagement) to .25 (Defensive vs.
Cool-Headed), averaging .20 across scales. These results provide support for the
psychometric qualities of the post-decision scales.

The following section presents evidence for the parallel forms used to test numerical and
verbal information processing.

2.5 Parallel Forms Evidence for Numerical and Verbal Scales

As previously noted, we developed two parallel forms to assess numerical and verbal
information processing. When using parallel forms, it is necessary to demonstrate that they
are parallel so that participants are likely to receive the same score regardless of which form
they complete.

To ensure that the forms of our numerical and verbal assessments would be parallel, we
first developed parallel items. For the verbal section, both forms rely on identical items that
are presented in a different order across forms within blocks of five. This ensures that
participants view the same items, and helps maintain item security because participants
cannot simply memorize one set of correct responses. We will continue to create additional
parallel forms as needed using this same process in the future. For the numerical section,
we created one set of items and slightly modified the numbers and/or patterns in each item
so that arriving at the correct answer across forms required the same logic and steps. We
created parallel distractor responses using this same approach, but presented distractors in
a different order to ensure that participants could not simply memorize one set of correct
responses.

To examine the effectiveness of these efforts, we compared descriptive statistics and


internal consistency estimates of reliability for each set of parallel forms, and conducted
independent samples t-tests on mean scores from our research sample. Tables 2.2 and 2.3
present these results.

Table 2.2
Parallel Forms Evidence for Numerical Information Processing
t-test
Numerical Form N Min Max M SD Skew Kurt
t df
Form 1 237 2 15 7.44 2.42 .18 -.23 .64
1.01 501 .31
Form 2 266 0 15 7.22 2.56 -.10 .22 .63
Note: N = Sample size; Min = Minimum score; Max = Maximum score; M = Mean; SD = Standard deviation;
Skew = Skewness statistic; Kurt = Kurtosis statistic; = Cronbachs alpha internal consistency reliability; t =
t-statistic from independent samples t-test; df = degrees of freedom; = significance level for t-statistic.

19

2. INVENTORY CONSTRUCTION, RELIABILITY, & STRUCTURAL PSYCHOMETRICS

Table 2.3
Parallel Forms Evidence for Verbal Information Processing
t-test
Verbal Form N Min Max M SD Skew Kurt
t df
Form 1 64 8 35 20.58 6.34 .05 -.42 .92
-.47 133 .64
Form 2 71 8 37 21.08 6.10 .54 .46 .92
Note: N = Sample size; Min = Minimum score; Max = Maximum score; M = Mean; SD = Standard deviation;
Skew = Skewness statistic; Kurt = Kurtosis statistic; = Cronbachs alpha internal consistency reliability; t =
t-statistic from independent samples t-test; df = degrees of freedom; = significance level for t-statistic.

These results reveal no significant differences in scale means, variance, or distributions,


indicating that the small differences that exist across forms for both assessments are simply
the result of sampling error. Therefore, no single form of the numerical and verbal
assessments offers participants an advantage, and the use of parallel forms helps maintain
assessment security.

2.6 Intercorrelations Between Hogan Judgment Model Scales

We computed correlations between the scales using the global sample of executives,
managers, and other high-level professionals previously described. Table 2.4 presents
these correlations.

Table 2.4
Correlations between Hogan Judgment Model Scales
Scale 1 2 3 4 5 6 7 8
1. Numerical Info. Processing 1.00 .45** .17** .10* -.16** .07 -.10* .10*
2. Verbal Info. Processing 1.00 .10* .06 -.10* .01 -.07 .08
3. Threat vs. Reward 1.00 .45** -.19** .13** -.36** .34**
4. Tactical vs. Strategic 1.00 -.08 .11* -.67** .47**
5. Data-Driven vs. Intuitive 1.00 -.09* .11* -.12**
6. Defensive vs. Cool-Headed 1.00 .00 .16**
7. Denial vs. Acceptance 1.00 -.32**
8. Superficial vs. Genuine 1.00
Note: N = 502; *Correlation is significant at .05 level; **Correlation is significant at .01 level.

Results reveal five points worth discussion. First, the numerical and verbal information
processing scales correlate at .45, indicating they are conceptually-related, yet independent,
components of cognitive ability. Second, the magnitudes of correlations between the
cognitive and non-cognitive scales of the model range from -.16 to .17, suggesting little
connection between these components. Third, correlations between pre-decision scales
(scales 3-5) range from -.19 to .45. Among these scales, the relationship between the
Threat Avoidance vs. Reward Seeking and Tactical vs. Strategic Thinking scales stands out.
However, this association makes intuitive sense, because organizational leaders often focus
on long-term rewards to succeed.

Fourth, correlations between post-decision scales (scales 6-8) range from -.32 to .16. Again,
this pattern of relationships suggests that these dimensions are moderately related, yet
measure independent constructs. Finally, correlations between the pre-decision and post-

20

2. INVENTORY CONSTRUCTION, RELIABILITY, & STRUCTURAL PSYCHOMETRICS

decision scales range between -.67 and .47, suggesting (as one would expect) that peoples
decision-making tendencies are somewhat related to how they react to feedback about
failed decisions. Specifically, the -.67 correlation between the Tactical vs. Strategic Thinking
scale and the Denial vs. Acceptance scale suggests that strategic decision-makers tend to
make big picture decisions and move on, whereas tactical decision-makers may be more
interested in revisiting their decisions and diving into the details of feedback.

Although the scales in our model are moderately correlated results in Table 2.4 suggest that
the scales represent unique and independent predictors of business judgment.

2.7 Structural Psychometrics of the Pre- and Post-Decision Scales

Factor analysis is a statistical methodology designed to account for the relationships


between many variables using a fewer number of factors. A factor represents something
shared in common by different variables; it is a linear combination of items which together
measure a single construct. Thus, this technique allows us to examine whether responses
to different items cluster together into meaningful factors and make sense of the many
relationships between individual assessment items.

We conducted an exploratory Principal Components Analysis (PCA) to identify underlying


factors of the pre- and post-decision scales. We used a sample of 872 employed adults.
Table 2.5 provides age, gender, and racial/ethnic data for this sample. Because the pre-
and post-decision scales target two different contexts, we conducted separate PCAs for pre-
and post-decision scales. We chose the number of components to extract based on the size
of the Eigenvalues and an examination of several alternate solutions. Finally, we refined the
components using orthogonal Varimax rotation to find the most economical solution with the
goal of associating each item to one factor.

Table 2.5
Sample Ethnic Composition by Age and Gender
Age Under 40 40 and Over
Gender Male Female Male Female
Race/Ethnicity N % N % N % N %
Two or More Races 3 0.34 7 0.80 0 0.00 0 0.00
Black/African-American 12 1.38 18 2.06 1 0.11 4 0.46
Hispanic/Latino 14 1.61 13 1.49 2 0.23 2 0.23
Asian 28 3.21 14 1.61 3 0.34 0 0.00
American Indian/Alaska Native 2 0.23 4 0.46 4 0.46 1 0.11
White 198 22.71 178 20.41 57 6.54 56 6.42
Not Indicated 2 0.23 10 1.15 0 0.00 1 0.11
Native Hawaiian/Other Pacific Islander 2 0.23 0 0.00 0 0.00 0 0.00
TOTAL 261 29.93 244 27.98 67 7.68 64 7.34
Note: 37 participants Under 40 and 1 participant 40 and Over did not identify their gender.

21

2. INVENTORY CONSTRUCTION, RELIABILITY, & STRUCTURAL PSYCHOMETRICS

2.7.1 Pre-Decision Scales

Table 2.6 presents PCA results for the pre-decision scales. To protect the proprietary nature
of our intellectual property, the table does not include text for specific assessment items.

Table 2.6
Varimax Rotated Factor Matrix for Pre-Decision Scales
Hogan Judgment Assessment Pre-Decision Scale Item Factor I Factor II Factor III
Threat Avoidance vs. Reward Seeking Item 1 -0.59
Threat Avoidance vs. Reward Seeking Item 2 -0.63
Threat Avoidance vs. Reward Seeking Item 3 -0.67
Threat Avoidance vs. Reward Seeking Item 4 -0.57
Threat Avoidance vs. Reward Seeking Item 5 -0.56
Threat Avoidance vs. Reward Seeking Item 6 -0.37
Threat Avoidance vs. Reward Seeking Item 7 -0.46
Threat Avoidance vs. Reward Seeking Item 8 0.37 0.47
Threat Avoidance vs. Reward Seeking Item 9 0.33
Threat Avoidance vs. Reward Seeking Item 10 0.44
Threat Avoidance vs. Reward Seeking Item 11 0.55
Threat Avoidance vs. Reward Seeking Item 12 0.47
Threat Avoidance vs. Reward Seeking Item 13 0.64
Threat Avoidance vs. Reward Seeking Item 14 0.66
Tactical vs. Strategic Thinking Item 1 0.77
Tactical vs. Strategic Thinking Item 2 0.59
Tactical vs. Strategic Thinking Item 3 0.67
Tactical vs. Strategic Thinking Item 4 -0.65
Tactical vs. Strategic Thinking Item 5 0.66
Tactical vs. Strategic Thinking Item 6 0.73
Tactical vs. Strategic Thinking Item 7 0.63
Tactical vs. Strategic Thinking Item 8 -0.55
Tactical vs. Strategic Thinking Item 9 -0.55
Tactical vs. Strategic Thinking Item 10 -0.64
Tactical vs. Strategic Thinking Item 11 0.44 0.31
Tactical vs. Strategic Thinking Item 12 0.77
Tactical vs. Strategic Thinking Item 13 -0.68
Data-Driven vs. Intuitive Item 1 -0.40
Data-Driven vs. Intuitive Item 2 -0.31
Data-Driven vs. Intuitive Item 3 0.69
Data-Driven vs. Intuitive Item 4 0.71
Data-Driven vs. Intuitive Item 5 0.76
Data-Driven vs. Intuitive Item 6 0.56
Data-Driven vs. Intuitive Item 7 0.73
Data-Driven vs. Intuitive Item 8 0.78
Data-Driven vs. Intuitive Item 9 0.68
Data-Driven vs. Intuitive Item 10 0.73
Data-Driven vs. Intuitive Item 11 0.59
Data-Driven vs. Intuitive Item 12 0.43

Results support the expected three-factor solution. Items for each scale loaded onto a
single factor (e.g., all Tactical/Strategic items loaded onto one factor). Only two items
demonstrated cross-loadings onto a second factor, but in both cases, the highest loading for
the item was still on the intended factor.

22

2. INVENTORY CONSTRUCTION, RELIABILITY, & STRUCTURAL PSYCHOMETRICS

2.7.2 Post-Decision Scales

Table 2.7 presents PCA results for the post-decision scales. Again, to protect the proprietary
nature of our intellectual property, the table does not include text for specific assessment
items.

Table 2.7
Varimax Rotated Factor Matrix for Post-Decision Scales
Hogan Judgment Assessment Post-Decision Scale Item Factor I Factor II Factor III
Defensive vs. Cool-Headed Item 1 0.56
Defensive vs. Cool-Headed Item 2 0.60
Defensive vs. Cool-Headed Item 3 0.64
Defensive vs. Cool-Headed Item 4 -0.73
Defensive vs. Cool-Headed Item 5 0.62
Defensive vs. Cool-Headed Item 6 -0.73
Defensive vs. Cool-Headed Item 7 0.53
Defensive vs. Cool-Headed Item 8 0.68
Defensive vs. Cool-Headed Item 9 0.76
Defensive vs. Cool-Headed Item 10 0.71
Defensive vs. Cool-Headed Item 11 0.43
Defensive vs. Cool-Headed Item 12 -0.65
Defensive vs. Cool-Headed Item 13 -0.55
Denial vs. Acceptance Item 1 0.57
Denial vs. Acceptance Item 2 0.56
Denial vs. Acceptance Item 3 0.66
Denial vs. Acceptance Item 4 0.62
Denial vs. Acceptance Item 5 0.51
Denial vs. Acceptance Item 6 0.59
Denial vs. Acceptance Item 7 0.54
Denial vs. Acceptance Item 8 0.68
Denial vs. Acceptance Item 9 0.63
Denial vs. Acceptance Item 10 0.53
Denial vs. Acceptance Item 11 0.55
Superficial vs. Genuine Engagement Item 1 0.56
Superficial vs. Genuine Engagement Item 2 0.59
Superficial vs. Genuine Engagement Item 3 0.52
Superficial vs. Genuine Engagement Item 4 0.63
Superficial vs. Genuine Engagement Item 5 0.52
Superficial vs. Genuine Engagement Item 6 0.63
Superficial vs. Genuine Engagement Item 7 0.44
Superficial vs. Genuine Engagement Item 8 0.44
Superficial vs. Genuine Engagement Item 9 0.39 -0.30
Superficial vs. Genuine Engagement Item 10 0.42
Superficial vs. Genuine Engagement Item 11 0.53
Superficial vs. Genuine Engagement Item 12 0.39 -0.35

Again, results generally support a three-factor solution. All but one item (#7 from the
Superficial vs. Genuine Engagement scale) loaded most strongly onto its intended factor,
and as with the pre-decision scales, there were only two items with cross-loadings. Although
adequate, these results demonstrate the need to continue examining the factor structure of
the scales as additional data become available and revising selected items as needed.

23

3. VALIDITY

3. Validity

3.1 Construct Validity

Validity concerns the meaning of scale scores. We believe the purpose of assessment is to
predict significant non-test outcomes. Following Gough (1975), we designed the Hogan
Judgment Report to predict how people process information, make decisions, and react to
feedback and coaching in work settings. Therefore, the validity of the Hogan Judgment
Report scales depends on empirical data that make sense given our theory of each scales
content (also see R. Hogan, Hogan, & Roberts, 1996). Such information includes
correlations with assessments measuring personality, values/ needs/motives/interests, and
cognitive ability.

The following section presents selected results from 16 such assessments. Appendix B
presents full correlation matrices between the Hogan Judgment Report scales and these
assessments.

3.1.1 Procedure

We obtained these data through Goldbergs (2008) longitudinal research with the Eugene-
Springfield Community Sample (ESCS). Goldberg recruited approximately 1,000 people to
participate in the project. However, only a portion of this sample completed both the
judgment assessment and the others we used to examine construct validity. We also
collected data using online, unproctored Internet testing. Subjects received compensation
for their participation. No participants took the assessment as part of high stakes testing
where results impacted hiring, promotion, or other personnel decisions.

In the following section, we report the sample composition for each assessment.

3.1.2 Samples and Instruments

HPI. First, we review judgment scale correlations with scales and subscales from the Hogan
Personality Inventory (HPI; R. Hogan & Hogan, 1995, 2007; see Table B.1). Ninety-eight
participants completed the Hogan Judgment Report and the HPI. The sample included 43
males and 55 females. Ages of subjects ranged from 29 years to 72 years with a mean of
46.31 years (SD = 8.00).

The HPI is a 206-item True/False measure of normal personality, which evolved from the
Five-Factor Model (FFM; De Raad & Perugini, 2002; Wiggins, 1996) and socioanalytic theory
(R. Hogan, 1983, 1991, 1996). Hogan and Holland (2003, p. 104) demonstrate that the
HPI is an adequate measure of the FFM, with median correlations with other FFM
inventories ranging from .30 to .69.

The global HPI normative group includes 144,877 working adults representing the global
workforce (R. Hogan & Hogan, 2007). The HPI contains seven primary scales and a validity
scale. In addition, a number of occupational scales are available for specialized
applications. The seven primary scales are Adjustment (ADJ), Ambition (AMB), Sociability

24

3. VALIDITY

(SOC), Interpersonal Sensitivity (INP), Prudence (PRU), Inquisitive (INQ), and Learning
Approach (LRN). The validity key (VAL) contains 14 items and detects careless or random
responding. The HPI technical manual (R. Hogan & Hogan, 2007) presents reliability,
confirmatory factor analysis, and validity. The Mental Measurements Yearbook (Lobello,
1996) and the British Psychological Societys Psychological Testing Centres test reviews
(British Psychological Society, 2007) also provide professional reviews of the HPI.

HDS. Second, we review judgment scale correlations with scales and subscales of the
Hogan Development Survey (HDS; R. Hogan & Hogan, 2009; Hogan Assessment Systems,
2014; see Table B.2). Eighty-five participants completed the Hogan Judgment Report and
the HDS. The sample included 41 males and 44 females. Ages of subjects ranged from 21
years to 67 years with a mean of 44.35 years (SD = 8.85).

The HDS concerns characteristics that can derail careers, relationships, and other
productive activities. The global HDS normative sample includes 67,614 working adults
representing the global workforce (R. Hogan & Hogan, 2009). The HDS contains 11 primary
scales: Excitable (EXC), Skeptical (SKE), Cautious (CAU), Reserved (RES), Leisurely (LEI),
Bold (BOL), Mischievous (MIS), Colorful (COL), Imaginative (IMA), Diligent (DIL), and Dutiful
(DUT). These scales assess dysfunctional dispositions that emerge when people stop
considering how their actions affect others. Over time, these dispositions create a persons
reputation and can impede job performance and career success.

The HDS is not a medical or clinical assessment. It does not measure personality disorders,
which are manifestations of mental disorder. Instead, the HDS assesses self-defeating
expressions of normal personality. The DSM-5 (American Psychiatric Association, 2013
p.647) makes this same distinction between behavioral traits and disorders self-defeating
behaviors, such as those predicted by the HDS, come and go depending on the context. In
contrast, personality disorders are enduring and pervasive across contexts.

The HDS technical manual (R. Hogan & Hogan, 2009) and the HDS Form 5 technical
supplement (Hogan Assessment Systems, 2014) provide more details about the reliability,
validity, factor structure, and norm development of the assessment.

CPI. Third, we review judgment scale correlations with scales from the California
Psychological Inventory (CPI; Gough, 1996; see Table B.3). One hundred and ninety-five
participants completed the Hogan Judgment Report and the CPI. The sample included 88
males and 107 females. Ages of subjects ranged from 21 years to 72 years with a mean of
45.71 years (SD = 8.78).

The CPI is a 434-item True/False inventory of personality. The CPI is normed on 52 male
and 42 female samples, including high school, college, graduate, professional school, and
occupational samples. The CPI contains 20 folk and 3 vector scales. The 20 folk scales are
Dominance (Do), Capacity for Status (Cs), Sociability (Sy), Social Presence (Sp), Self-
acceptance (Sa), Independence (In), Empathy (Em), Responsibility (Re), Socialization (So),
Self-control (Sc), Good Impression (GI), Communality (Cm), Well-being (Wb), Tolerance (To),
Achievement via Conformance (Ac), Achievement via Independence (Ai), Intellectual
Efficiency (Ie), Psychological-mindedness (Py), Flexibility (Fx), and Femininity/Masculinity

25

3. VALIDITY

(F/M). The three vector scales are Externality/Internality (v.1), Norm-doubting/Norm-


favoring (v.2), and Ego-integration (v.3). The CPI technical manual (Gough, 1996) provides
details about reliability, factor analysis, and validity of the assessment.

NEO PI-R. Fourth, we review judgment scale correlations with scales and facets from the
NEO PI-R (Costa & McCrae, 1992; see Table B.4). One hundred and eighty-two participants
completed the Hogan Judgment Report and the NEO PI-R. The sample included 82 males
and 100 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.96
years (SD = 8.89).

The NEO PI-R is a 240-item True/False measure of personality. Specifically, it measures five
major dimensions of personality, as well as important facets of each domain with
applicability in both clinical and research domains. The NEO PI-R norming sample includes
1,000 adults, representing a stratification of the U.S. population based on race and age
(Costa & McCrae, 1992). The NEO PI-R contains 5 domain scales and 30 facet scales. The
5 domains are Neuroticism (N), Extraversion (E), Openness (O), Agreeableness (A), and
Conscientiousness (C). Six facet scales fall within each domain. The Neuroticism facets are
Anxiety (N1), Angry Hostility (N2), Depression (N3), Self-Consciousness (N4), Impulsiveness
(N5), and Vulnerability (N6). The Extraversion facets are Warmth (E1), Gregariousness (E2),
Assertiveness (E3), Activity (E4), Excitement-Seeking (E5), and Positive Emotions (E6). The
Openness facets are Fantasy (O1), Aesthetics (O2), Feelings (O3), Actions (O4), Ideas (O5),
and Values (O6). The Agreeableness facets are Trust (A1), Straightforwardness (A2),
Altruism (A3), Compliance (A4), Modesty (A5), and Tender-Mindedness (A6). The
Conscientiousness facets are Competence (C1), Order (C2), Dutifulness (C3), Achievement
Striving (C4), Self-Discipline (C5), and Deliberation (C6). The NEO PI-R technical manual
(Costa & McCrae, 1992) provides details about the reliability, factor analysis, and validity of
the assessment.

IPIP. Fifth, we review judgment scale correlations with the International Personality Item
Pool (IPIP; Goldberg, Johnson, Eber, Hogan, Ashton, Cloninger, & Gough, 2006; see Table
B.5). One hundred and sixty-nine participants completed the Hogan Judgment Report and
the IPIP. The sample included 76 males and 93 females. Ages of subjects ranged from 22
years to 72 years with a mean of 45.86 years (SD = 8.71).

The International Personality Item Pool (IPIP; Goldberg, 1999; Goldberg, et al., 2006) is an
online pool of over 2,000 personality assessment items. The purpose of IPIP is to
continuously develop and refine personality inventories. The IPIP is available for anyone to
contribute and/or use. Currently, users can create 269 scales using the available items.
The IPIP and corresponding scales are updated regularly to use new and refined items. We
presented correlations using the following scales: Extraversion (EXT), Agreeableness (AGR),
Conscientiousness (CON), Emotional Stability (EMS), and Intellect/Imagination (I/I).
Goldberg et al. (2006) provides technical features of the IPIP, including norm samples, scale
construction, and validity indices. The IPIP website (http://ipip.ori.org) provides additional
information.

16PF. Sixth, we review judgment scale correlations with scales from the Sixteen Personality
Factor Questionnaire (16PF; Conn & Rieke, 1994; Russell & Karol, 2002; see Table B.6).

26

3. VALIDITY

One hundred and ninety-seven participants completed the Hogan Judgment Report and the
16PF. The sample included 89 males and 108 females. Ages of subjects ranged from 22
years to 72 years with a mean of 45.55 years (SD = 8.39).

The 16PF is a 185-item measure of normal personality, whose foundation is based on factor
analyzing all English-language adjectives describing human behavior. The 16PF norming
sample includes 10,261 adults representing a stratification of the U.S. adult population.
The assessment contains 16 bipolar scales with meaning for both high and low scale scores.
In addition, there are five global factor scales and an impression management index
assessing social desirability. The 16 factor scales are Factor A: Warmth, Factor B:
Reasoning, Factor C: Emotional Stability, Factor E: Dominance, Factor F: Liveliness, Factor G:
Rule-Consciousness, Factor H: Social Boldness, Factor I: Sensitivity, Factor L: Vigilance,
Factor M: Abstraction, Factor N: Privateness, Factor O: Apprehension, Factor Q1: Openness
to Change, Factor Q2: Self-Reliance, Factor Q3: Perfectionism, and Factor Q4: Tension. The
five global factor scales are Extraversion (EX), Anxiety (AX), Tough-mindedness (TM),
Independence (IN), and Self-Control (SC). Conn and Rieke (1994) provide information on
the technical features of the assessment, including reliability, item analysis, factor analysis,
and validity.

TCI. Seventh, we review judgment scale correlations with scales from the Temperament and
Character Inventory (TCI; see Table B.7). Two hundred and three participants completed the
Hogan Judgment Report and the TCI. The sample included 91 males and 112 females.
Ages of subjects ranged from 21 years to 72 years with a mean of 45.45 years (SD = 8.71).

The TCI is a 295-item adaptation of Cloninger, Przybeck, Svrakic, and Wetzel's (1994) multi-
level personality inventory, with 35 lower-level scales subsumed by 7 higher-level constructs.
The four temperament dimensions are: Novelty Seeking (NS; Exploratory excitability vs. Stoic
rigidity, Impulsiveness vs. Reflection, Extravagance vs. Reserve, Disorderliness vs.
Regimentation); Harm Avoidance (HA; Anticipatory worry and pessimism vs. Uninhibited
optimism, Fear of uncertainty, Shyness with strangers, Fatigability and asthenia); Reward
Dependence (RD; Sentimentality, Openness to warm communication vs. Aloofness,
Attachment, Dependence); and Persistence (PS; Eagerness of effort vs. Laziness, Work-
hardened vs. Spoiled, Ambitious vs. Underachieving, Perfectionist vs. Pragmatist). The three
character dimensions are: Self-Directedness (SD; Responsibility vs. Blaming, Purposefulness
vs. Lack of goal-orientation, Resourcefulness, Self-acceptance vs. Self-striving, Enlightened
Second Nature); Cooperativeness (CO; Social acceptance vs. Social intolerance, Empathy vs.
Social disinterest, Helpfulness vs. Unhelpfulness, Compassion vs. Revengefulness, Pure-
hearted Conscience vs. Self-serving advantage); and Self-Transcendence (ST; Self-forgetful
vs. Self-conscious experience, Transpersonal identification vs. Self-differentiation, Spiritual
acceptance vs. Rational materialism; Enlightened vs. Objective, Idealistic vs. Practical). The
TCI website (https://tci.anthropedia.org/en/) provides information on the technical features
of the assessment.

MPQ. Eighth, we review judgment scale correlations with scales from the Multidimensional
Personality Questionnaire (MPQ; Tellegen & Waller, 2008; see Table B.8). Two hundred and
five participants completed the Hogan Judgment Report and the MPQ. The sample included

27

3. VALIDITY

93 males and 112 females. Ages of subjects ranged from 21 years to 72 years with a mean
of 45.58 years (SD = 8.65).

The MPQ contains 11 primary trait dimension scales. Most of the 276 MPQ items use a
True/False response format. The norming sample comes from the Minnesota Twins
Registry, which is a community sample of 1,350 individuals between 20 and 60 years of
age. The sample includes 675 males and 675 females. The 11 primary trait scales of the
MPQ include Wellbeing (WB), Social Potency (SP), Achievement (AC), Social Closeness (SC),
Stress Reaction (SR), Aggression (AG), Alienation (AL), Control (CO), Harm Avoidance (HA),
Traditionalism (TR), and Absorption (AB). Tellegen and Waller (2008) provide information on
the technical features of the assessment, including item analysis, reliability, factor analysis,
and validity.

6FPQ. Ninth, we review judgment scale correlations with factor and facet scores from the
Six Factor Personality Questionnaire (6FPQ; Jackson, Paunonen, Fraboni, & Goffin, 1996;
see Table B.9). Two hundred participants completed the Hogan Judgment Report and the
6FPQ. The sample included 90 males and 110 females. Ages of subjects ranged from 21
years to 72 years with a mean of 45.44 years (SD = 8.53).

The 6FPQ extends FFM personality dimensions to measure six dimensions: Extraversion,
Agreeableness, Independence, Openness to Experience, Methodicalness, and
Industriousness. Specifically, the 6FPQ divides Conscientiousness into Methodicalness and
Industriousness, and redefines Neuroticism as the opposite pole of the dimension,
Independence. The norm group includes a random sample of 483 males and 584 females
in the U.S. and Canada. The test publisher also provides a norm sample of 4,040 job
applicants to support the workforce application of the assessment.

The 6FPQ includes six factor scales, each with three subscales: Extraversion (Affiliation;
Dominance; Exhibition); Agreeableness (Abasement; Even-tempered; Good-natured);
Independence (Autonomy; Individualism; Self-reliance); Openness to Experience (Change;
Understanding; Breadth of Interest); Methodicalness (Cognitive Structure; Deliberateness;
Order); and Industriousness (Achievement; Endurance; Seriousness). The 6FPQ technical
manual (Jackson, 1984) and relevant research studies (e.g., Jackson, Paunonen, Fraboni, &
Goffin, 1996) provide information on the technical features of the assessment, including
reliability, factor analysis, and validity.

PRS. Tenth, we review judgment scale correlations with personality content from the
Personal Reactions Survey (PRS; see Table B.10). Two hundred and eight participants
completed the Hogan Judgment Report and the PRS. The sample included 94 males and
114 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.46
years (SD = 8.68).

The PRS includes 192 items from the HEXACO Personality Inventory (Lee & Ashton, 2004),
20 items from four Behavioral Activation System (BAS) and Behavioral Inhibition System
(BIS) scales from Carver and White (1994), and 23 items selected by Jackson (2003) as
markers of the major factors in the Gray-Wilson Personality Questionnaire (Wilson, Barrett, &

28

3. VALIDITY

Gray, 1989). However, to validate the Hogan Judgment Report scales, our analyses focused
on the BAS, BIS, and HEXACO scales.

MVPI. Eleventh, we review judgment scale correlations with scales and themes from the
Motives, Values, Preferences Inventory (MVPI; J. Hogan & Hogan, 2010; see Table B.11).
Ninety-nine participants completed the Hogan Judgment Report and the MVPI. The sample
included 44 males and 55 females. Ages of subjects ranged from 30 years to 68 years with
a mean of 45.31 years (SD = 8.16).

The MVPI is a 200-item measure of values and preferences which assesses individual
motives and evaluates person-organization fit. The global MVPI normative sample includes
48,267 working adults representing the global workforce (J. Hogan & Hogan, 2010). The
MVPI contains 10 primary scales: Aesthetics (AES), Affiliation (AFF), Altruism (ALT),
Commerce (COM), Hedonism (HED), Power (POW), Recognition (REC), Science (SCI), Security
(SEC), and Tradition (TRA). Each scale also reflects five themes: Lifestyles, Beliefs,
Occupational Preferences, Aversions, and Preferred Associations. The MVPI technical
manual (J. Hogan & Hogan, 2010) provides details about item content, reliability, factor
analysis, and validity of the instrument.

CISS. Twelfth, we review judgment scale correlations with interest and skill scales from the
Campbell Interest and Skill Survey (CISS; Campbell, Hyne, & Nilsen, 1992; see Table B.12).
One hundred and sixty-six participants completed the Hogan Judgment Report and the CISS.
The sample included 76 males and 90 females. Ages of subjects ranged from 21 years to
72 years with a mean of 45.03 years (SD = 8.80).

The CISS maps self-reported skills and interests to the occupational world to provide
individuals with career guidance. It is intended for use with most adults and students as
young as 15. This survey contains 200 interest items and 120 skill items measured on a
six-point scale. The interest scales provide an indicator for the strength of attraction to
occupational areas, while the skills scales are an estimate of competence.

Overall, there are seven major orientation scales (i.e., Influencing, Organizing, Helping,
Creating, Analyzing, Producing, and Adventuring), which indicate attraction to and
confidence in each orientation. Individuals receive both interest and skill scores for each
orientation, and can score either high or low on both, providing four possible interest and
skill combinations for each orientation. There are also 29 basic interest and skill subscales
for the orientations, covering specific topics such as public speaking or mathematics. The
CISS norming sample includes 5,000 people from over 60 different occupations. Campbell,
Hyne, and Nilsen (1992) provide information on the technical features of the assessment,
including item analysis, scale construction, reliability, and validity.

JPI-R. Thirteenth, we review judgment scale correlations with scales from the Jackson
Personality Inventory - Revised (JPI-R; Jackson, 1994; see Table B.13). Two hundred and
seven participants completed the Hogan Judgment Report and the JPI-R. The sample
included 94 males and 113 females. Ages of subjects ranged from 21 years to 72 years
with a mean of 45.45 years (SD = 8.69).

29

3. VALIDITY

The JPI-R is a 300-item True/False measure of personality concerning individuals


interpersonal patterns of interaction, cognitive styles, and value orientations. The JPI-R is
normed on four different populations: high school students, blue-collar workers, executives,
and adult college students. The college, blue collar, and executive norm groups are
combined to form an overall norm group of 1,436 individuals (Jackson, 1994). The JPI-R
contains 15 content scales: Complexity (Cpx), Breadth of Interest (Bdi), Innovation (Inv),
Tolerance (Tol), Empathy (Emp), Anxiety (Axy), Cooperativeness (Cpr), Sociability (Soc), Social
Confidence (Scf), Energy Level (Enl), Social Astuteness (Sas), Risk Taking (Rkt), Organization
(Org), Traditional Values (Trv), and Responsibility (Rsy). These scales are grouped into five
meaningful clusters: Analytical, Emotional, Extroverted, Opportunistic, and Dependable.
Jackson (1994) provides information on the technical features of the assessment, including
item analysis, reliability, factor analysis, and validity.

HBRI. Fourteenth, we review judgment scale correlations with scales from the Hogan
Business Reasoning Inventory (HBRI; R. Hogan, Barrett, & Hogan, 2007; see Table B.14).
Two hundred and five participants completed the Hogan Judgment Report and the HBRI.
The sample included 94 males and 111 females. Ages of subjects ranged from 21 years to
72 years with a mean of 45.44 years (SD = 8.74).

The HBRI is a 24-item measure of cognitive skills and business reasoning, intended for use
with college-educated (i.e., bachelors degree and above) managers and professionals.
Items reflect cognitive tasks with content similar to that used in actual business operations.
The HBRI norming sample includes 2,484 adults who completed the assessment as part of
applicant screening, employee development, or research purposes (R. Hogan, Barrett, &
Hogan, 2007). The HBRI includes strategic reasoning and tactical reasoning scales, which
are combined to examine overall critical reasoning. The HBRI technical manual (R. Hogan,
Barrett, & Hogan, 2007) provides information on the technical features of the test, including
reliability, scale construction, factor analysis, and validity.

SPA. Fifteenth, we review judgment scale correlations with cognitive content from the Skills,
Possessions, and Abilities instrument (SPA; see Table B.15). Two hundred and eleven
participants completed the Hogan Judgment Report and the SPA. The sample included 95
males and 116 females. Ages of subjects ranged from 21 years to 72 years with a mean of
45.51 years (SD = 8.72). Although the SPA instrument included a broad range of content
(e.g., computer usage and skill, number of various types of possessions owned, cultural
knowledge, self-ratings of various talents), we were mainly interested in how the cognitive
scales from the Hogan Judgment Report (i.e., Verbal and Numerical Information Processing)
related to assessments of verbal and mathematical ability in the SPA.

BRI. Finally, we review judgment scale correlations with cognitive content from the
Behavioral Report Inventory (BRI; see Table B.16). Two hundred and five participants
completed the Hogan Judgment Report and the BRI. The sample included 93 males and
112 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.32
years (SD = 8.63). Although the BRI included a broad range of content (e.g., 400 activity
descriptions, 49 eating habits and practices, 24 symptoms of depression, 31 kinds of
dissociative experiences), we were mainly interested in how the cognitive scales from the

30

3. VALIDITY

Hogan Judgment Report (i.e., Verbal and Numerical Information Processing) related to
similar content on cognitive activities included in the BRI.

3.1.3 Results of Scale to Scale Correlates

In the following section, we present selected statistically significant correlations between


Hogan Judgment Report scales and those from the assessments previously described to
illustrate relationships with behaviors that are conceptually aligned with the content of each
scale. To examine full correlation matrices, please refer to Appendix B.

Threat Avoidance vs. Reward Seeking. Table 3.1 provides selected significant correlations
between the Threat Avoidance vs. Reward Seeking scale and scales from other
assessments. Negative correlations indicate stronger relationships with Threat Avoidance
whereas positive correlations indicate stronger relationships with Reward Seeking.

Table 3.1
Construct Validity Evidence for Threat Avoidance vs. Reward Seeking Scale
Assessment Scale Threat Avoidance vs. Reward Seeking
HDS Cautious -.35**
MVPI Security -.57**
IPIP Conscientiousness -.31**
TCI Fear of Uncertainty -.50**
MPQ Harm Avoidance -.36**
PRS Fearfulness -.44**
HPI Ambition .34**
Sociability .39**
Inquisitive .57**
HDS Mischievous .42**
Colorful .47**
MVPI Power .27**
CPI Social Presence .39**
Capacity for Status .36**
NEO PI-R Openness/Actions .39**
16PF Openness to Change .36**
CISS Adventuring Interests .31**
Adventuring Skills .33**
JPI-R Risk Taking .57**
6 FPQ Change Facet Scale .37**
Openness to Experience .34**
PRS Openness to Experience .38**
Unconventionality .34**
Note. ** Correlation is significant at .01 level; Negative correlations indicate relationships with Threat
Avoidance; Positive correlations indicate relationships with Reward Seeking.

Across assessments, four themes emerge. First, individuals who are open to change and
new experiences tend to seek rewards when making decisions. Significant positive
correlations between Reward Seeking and HPI Inquisitive (.57), NEO PI-R Openness/Actions
(.39), 16PF Openness to Change (.36), CISS Adventuring Interests (.31) and Skills (.33),
6FPQ Change (.37) and Openness to Experience (.34), and PRS Openness to Experience
(.38) and Unconventionality (.34) scales underscore these tendencies. Conversely, persons
who fear change tend to adopt a threat avoidant perspective when making decisions.

31

3. VALIDITY

Significant negative correlations with HDS Cautious (-.35), MVPI Security (-.57), TCI Fear of
Uncertainty (-.50), MPQ Harm Avoidance (-.36), and PRS Fearfulness (-.44) scales suggest
that decision-makers who are open to novel experience tend to be more tolerant of
uncertainty and driven by potential rewards, whereas individuals who are unassertive and
fear the unknown tend to focus on potential risks.

Second, individuals who enjoy power and status tend to adopt a reward-based goal
orientation when making decisions. Significant positive correlations with HPI Ambition (.34),
MVPI Power (.27), and CPI Capacity for Status (.36) indicate that motivation to achieve,
influence, and obtain status may drive decisions that focus on rewards.

Third, reward-seeking decision-makers also appear to be sociable and attention seeking, as


evidenced by significant positive correlations with HPI Sociability (.39), HDS Colorful (.47),
and CPI Social Presence (.39). These results indicate that individuals who enjoy social
interaction may focus on rewards associated with potential public recognition.

Fourth, individuals who are willing to take risks also tend to seek rewards. Significant
positive correlations with HDS Mischievous (.42), JPI-R Risk Taking (.57), and the significant
negative correlation with IPIP Conscientiousness (-.31), suggest that risk-takers tend to
focus on rewards, whereas conscientious individuals focus on potential losses.

Tactical vs. Strategic Thinking. Table 3.2 provides selected significant correlations between
the Tactical vs. Strategic Thinking scale and scales from other assessments. Negative
correlations indicate relationships with Tactical Thinking, whereas positive correlations
indicate relationships with Strategic Thinking.

32

3. VALIDITY

Table 3.2
Construct Validity Evidence for Tactical vs. Strategic Thinking Scale
Assessment Scale Tactical vs. Strategic Thinking
HDS Skeptical -.26**
Cautious -.62**
Reserved -.31**
NEO PI-R Self-Consciousness -.42**
16PF Apprehension -.27**
MVPI Security -.28**
HPI Ambition .66**
HDS Colorful .44**
MVPI Power .41**
CPI Dominance .69**
Leadership .57**
NEO PI-R Extraversion/Assertiveness .67**
Conscientiousness/Achievement Striving .40**
IPIP Extraversion .47**
16PF Social Boldness .59**
CISS Influencing Interests .46**
Influencing Skills .47**
TCI Resourcefulness .48**
MPQ Social Potency .62**
JPI-R Social Confidence .66**
6FPQ Extraversion .58**
Dominance .52**
PRS Social Boldness .69**
Extraversion .55**
Note. ** Correlation is significant at .01 level; Negative correlations indicate relationships with Tactical
Thinking; Positive correlations indicate relationships with Strategic Thinking.

Three themes are associated with this scale. First, people who are ambitious and driven to
achieve tend to focus on larger, future-oriented issues. Significant positive correlations with
HPI Ambition (.66), MVPI Power (.41), CPI Dominance (.69) and Leadership (.57), NEO PI-R
Conscientiousness/Achievement Striving (.40), 16PF Social Boldness (.59), CISS Influencing
Interests (.46) and Skills (.47), 6FPQ Dominance (.52), and PRS Social Boldness (.69) scales
reflect this tendency.

Second, individuals who are skilled at and enjoy social interaction tend to focus on big-
picture issues over the long term. These tendencies are borne out by significant positive
correlations with HDS Colorful (.44), NEO PI-R Extraversion/Assertiveness (.67), IPIP
Extraversion (.47), MPQ Social Potency (.62), JPI-R Social Confidence (.66), 6FPQ
Extraversion (.58), and PRS Extraversion (.55).

Third, individuals who tend to be mistrustful, cautious, and apprehensive tend to take a
practical and detailed approach to decision-making. Significant negative correlations with
HDS Cautious (-.62), NEO PI-R Self-Consciousness (-.42), 16PF Apprehension (-.27), and
MVPI Security (-.28) suggest that such individuals cling to relevant details as a means of
coping with mistrust or fear. Conversely, the significant positive correlation with TCI
Resourcefulness (.48) suggests that strategic thinkers are willing to make decisions and
able to leverage available resources to inform those decisions.

33

3. VALIDITY

Data-Driven vs. Intuitive Decisions. Table 3.3 provides selected significant correlations
between the Data-Driven vs. Intuitive Decisions scale and scales from other assessments.
Negative correlations indicate relationships with Data-Driven Decisions, whereas positive
correlations indicate relationships with Intuitive Decisions.

Table 3.3
Construct Validity Evidence for Data-Driven vs. Intuitive Decisions Scale
Assessment Scale Data-Driven vs. Intuitive Decisions
HPI Inquisitive -.47**
Learning Approach -.33**
MVPI Scientific -.67**
IPIP Intellect -.29**
CISS Analyzing Interests -.57**
Analyzing Skills -.46**
PRS Inquisitiveness -.52**
CPI Intellectual Efficiency .29**
MPQ Traditionalism .25**
16PF Social-Boldness .20**
JPI-R Cooperativeness .21**
Traditional Values .32**
Note. ** Correlation is significant at .01 level; Negative correlations indicate relationships with Data-Driven
Decisions; Positive correlations indicate relationships with Intuitive Decisions.

Two themes emerge from these results for this scale. First, individuals who are naturally
curious and analytical tend to make decisions based on a careful review of relevant data.
Significant correlations with HPI Learning Approach (-.33), MVPI Scientific (-.67), IPIP
Intellect (-.29), CISS Analyzing Interests (-.57) and Skills (-.46), and PRS Inquisitiveness (.-
.52) scales reflect these tendencies. Decision-makers who like to learn and remain current
with industry trends, and who are objective and analytical prefer to make decisions based
on data.

Second, socially bold leaders and those who value the past tend to make decisions based
on career experience and intuition. Positive correlations with 16PF Social-Boldness (.20)
and traditional value scales from the MPQ (.25) and JPI-R (.32) underscore this fact. The
correlation with CPI Intellectual Efficiency (.29) provides additional evidence that intuitive
decision-makers prefer to reach quick decisions guided by previous experience.

Defensive vs. Cool-Headed. Table 3.4 presents selected significant correlations between
the Defensive vs. Cool-Headed scale and those from other assessments. Negative
correlations indicate Defensive reactions to negative feedback, whereas positive
correlations indicate Cool-Headed reactions to feedback.

34

3. VALIDITY

Table 3.4
Construct Validity Evidence for Defensive vs. Cool-Headed Scale
Assessment Scale Defensive vs. Cool-Headed
HDS Excitable -.47**
Skeptical -.45**
NEO PI-R Neuroticism -.49**
MPQ Stress Reaction -.51**
JPI-R Anxiety -.49**
HPI Adjustment .76**
Prudence .48**
CPI Well-Being .53**
Self-Control .47**
Amicability .48**
Work Orientation .46**
IPIP Emotional Stability .46**
16PF Emotional Stability .38**
6FPQ Even-Tempered .59**
PRS Patience .54**
Note. ** Correlation is significant at .01 level; Negative correlations indicate relationships with Defensive
reactions; Positive correlations indicate relationships with Cool-Headed reactions.

Three themes are of note with this scale. First, emotionally stable individuals tend to be
cool-headed when receiving feedback about their mistakes. This is seen in the positive
correlations with HPI Adjustment (.76), IPIP Emotional Stability (.46), 16PF Emotional
Stability (.38), 6FPQ Even-Tempered (.59), and PRS Patience (.54) scales. Conversely,
individuals who are less emotionally poised tend to take a more defensive posture in
reacting to feedback about their mistakes. Negative correlations with HDS Excitable (-.47),
NEO PI-R Neuroticism (-.49), MPQ Stress Reaction (-.51), and JPI-R Anxiety (-.49) scales
reflect these tendencies.

Second, individuals who are dependable, self-controlled, and focused on the task at hand
tend to remain calm in the face of negative feedback, as evidenced by positive correlations
with HPI Prudence (.48), CPI Self-Control (.47), Amicability (.48), and Work Orientation (.46)
scales. Because these individuals are normally disposed to take responsibility for their
actions, they remain calm when receiving negative feedback because they remain focused
on the work and how to make future improvements.

Third, individuals who tend to be critical of themselves and others naturally become
defensive when given negative feedback, as seen by the negative correlation with HDS
Skeptical (-.45). It is possible that these individuals are more concerned with finding flaws
in the feedback than with identifying errors in their decision-making, which may present
obstacles to learning from their mistakes.

Denial vs. Acceptance. Table 3.5 contains significant correlations between the Denial vs.
Acceptance scale and those from other assessments. Negative correlations indicate Denial,
whereas positive correlations indicate Acceptance.

35

3. VALIDITY

Table 3.5
Construct Validity Evidence for Denial vs. Acceptance Scale
Assessment Scale Denial vs. Acceptance
HPI Ambition -.58**
HDS Bold -.55**
MVPI Power -.58**
Recognition -.41**
CPI Dominance -.53**
NEO PI-R Extraversion/Assertiveness -.52**
Conscientiousness/Achievement Striving -.45**
IPIP Extraversion -.30**
16PF Dominance -.43**
Social Boldness -.37**
CISS Influencing Interests -.35**
Influencing Skills -.37**
MPQ Social Potency -.57**
Achievement -.36**
JPI-R Social Confidence -.49**
6FPQ Extraversion -.50**
PRS Extraversion -.42**
HDS Cautious .44**
NEO PI-R Self-Consciousness .22**
Vulnerability .25**
Agreeableness .29**
Compliance .25**
Modesty .35**
6FPQ Agreeableness .19**
Good-Natured .21**
PRS HEXACO Modesty .41**
HEXACO Flexibility .21**
HEXACO Honesty-Humility .25**
Note. ** Correlation is significant at .01 level; Negative correlations indicate relationships with Denial; Positive
correlations indicate relationships with Acceptance.

Three themes emerge regarding this scale. First, ambitious individuals who value power and
status tend to deny negative feedback. Negative correlations with HPI Ambition (-.58), HDS
Bold (-.55), MVPI Power (-.58) and Recognition (-.41), CPI Dominance (-.53), NEO PI-R
Conscientiousness/Achievement Striving (-.45), 16PF Dominance (-.43), CISS Influencing
Interests (-.35) and Skills (-.37), and MPQ Achievement (-.36) reflect these tendencies.
These results suggest that power seeking individuals tend to resist criticism, perhaps
because they see it as a challenge to their authority.

Outgoing and highly social individuals also tend to deny negative feedback. Negative
correlations with NEO PI-R Extraversion/Assertiveness (-.52), IPIP Extraversion (-.30), 16PF
Social Boldness (-.37), MPQ Social Potency (-.57), JPI-R Social Confidence (-.49), and
Extraversion scales from the 6FPQ (-.50) and PRS (-.42) suggest that socially skilled
individuals may view negative feedback as an assault on their social competence.

Third, individuals who are cautious, modest, and agreeable tend to accept negative
feedback about their mistakes. Significant positive correlations with HDS Cautious (.44),
NEO PI-R Compliance (.25), Agreeableness scales from the NEO PI-R (.29) and 6FPQ (.19),

36

3. VALIDITY

and Modesty scales from the NEO PI-R (.35) and PRS (.41) suggest that humble and
agreeable individuals are more likely to accept negative feedback than those who are bold
and socially dominant.

Superficial vs. Genuine Engagement. Table 3.6 provides selected correlations between the
Superficial vs. Genuine Engagement scale and those from other assessments. Negative
correlations indicate Superficial Engagement, whereas positive correlations indicate
Genuine Engagement.

Table 3.6
Construct Validity Evidence for Superficial vs. Genuine Engagement Scale
Assessment Scale Superficial vs. Genuine Engagement
HDS Excitable -.30**
Cautious -.42**
Leisurely -.32**
MVPI Security -.26**
NEO PI-R Neuroticism -.34**
Self-Consciousness -.41**
16PF Apprehension -.34**
JPI-R Cooperativeness -.33**
MPQ Stress Reaction -.30**
HPI Ambition .49**
Inquisitive .31**
Colorful .40**
CPI Tough-Mindedness .46**
Leadership .48**
TCI Resourcefulness .45**
MPQ Social Potency .26**
JPI-R Social Confidence .37**
6FPQ Dominance .38**
Extraversion .35**
PRS Social Boldness .39**
Note. ** Correlation is significant at .01 level; Negative correlations indicate relationships with Superficial
Engagement; Positive correlations indicate relationships with Genuine Engagement.

Across assessments, four themes emerge. First, achievement oriented individuals tend to
genuinely engage in feedback. Positive correlations with HPI Ambition (.49), CPI Tough-
Mindedness (.46) and Leadership (.48), TCI Resourcefulness (.45), and 6FPQ Dominance
(.38) scales support these tendencies, and suggest that decision-makers who are
competitive, leader-like, and willing to make tough decisions are willing to take on negative
feedback in order to improve future performance.

Second, individuals who are socially skilled are also willing to genuinely engage in negative
feedback. Positive correlations with HDS Colorful (.40), MPQ Social Potency (.26), JPI-R
Social Confidence (.37), 6FPQ Extraversion (.35), and PRS Social Boldness (.39) scales
indicate that individuals who are adept at interacting with others are willing to listen to
feedback in order to learn from their past mistakes.

Third, individuals who are emotionally volatile tend to superficially engage in feedback to
maintain positive social impressions and avoid a genuine confrontation with their problems.

37

3. VALIDITY

Negative correlations with HDS Excitable (-.30), NEO PI-R Neuroticism (-.34), and MPQ Stress
Reaction (-.30) suggest that these individuals avoid deeper exploration of their failures
because doing so is threatening.

Finally, individuals who value predictability and order or are cautious, apprehensive, and
cooperative may also superficially engage in negative feedback. Negative correlations with
HDS Cautious (-.42), NEO PI-R Self-Consciousness (-.41), 16PF Apprehension (-.34), JPI-R
Cooperativeness (-.33), and MVPI Security (-.26) underscore these points. In general,
individuals who lack poise are more likely to engage superficially compared to those who
remain calm and confident.

Openness to Feedback and Coaching. Table 3.7 provides selected correlations between the
overall Openness to Feedback and Coaching scale from the Hogan Judgment Report and
scales from other assessments.

Table 3.7
Construct Validity Evidence for Openness to Feedback and Coaching Scale
Assessment Scale Openness to Feedback and Coaching
HDS Excitable -.40**
Skeptical -.41**
NEO PI-R Neuroticism -.44**
Angry Hostility -.47**
MPQ Stress Reaction -.43**
PRS HEXACO Anxiety -.42**
MVPI Hedonism -.42**
Recognition -.45**
JPI-R Anxiety -.50**
HPI Adjustment .56**
CPI Self-Control .49**
Well-Being .51**
Tolerance .44**
Ego-Integration .44**
IPIP Emotional Stability .50**
6FPQ Even-Tempered .46**
Note. ** Correlation is significant at .01 level.

Three themes emerge with regard to this scale. First, emotionally stable individuals seem
most able to profit from coaching and learning from that process. Positive correlations with
HPI Adjustment (.56), CPI Self-Control (.49), IPIP Emotional Stability (.50), and 6FPQ Even-
Tempered (.46) scales reflect the beneficial nature of these tendencies.

Second and conversely, highly emotional individuals seem least receptive to feedback and
coaching around their past mistakes. Negative correlations with HDS Excitable (-.40), NEO
PI-R Neuroticism (-.44) and Angry Hostility (-.47), MPQ Stress Reaction (-.43), HEXACO
Anxiety from the PRS (-.42), and JPI-R Anxiety (-.50) underscore this point.

Third, individuals who do not place value in others, or who place excess value on leisure and
personal recognition, may encounter difficulties in remaining receptive to feedback and
coaching. Negative correlations with HDS Skeptical (-.41) and MVPI Hedonism (-.42) and

38

3. VALIDITY

Recognition (-.45) reflect the fact that trust in others, humility, and a business-like approach
are often needed to benefit from feedback and coaching.

Verbal Information Processing. Table 3.8 provides selected significant correlations between
scores on the Verbal Information Processing scale from the Hogan Judgment Report and
cognitive content from other assessments.

Table 3.8
Construct Validity Evidence for Verbal Information Processing Scale
Assessment Scale Verbal Informational Processing
SPA Verbal Ability .16*
BRI Reading Cluster .15*
Books Cluster .15*
HPI Learning Approach .32**
HBRI Tactical Thinking .45**
Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level.

Across assessments, Verbal Information Processing scale scores show significant positive
correlations with SPA Verbal Ability (.16), and BRI Reading (.15) and Books (.15) Clusters.
The strongest noted correlation for this scale is with HBRI Tactical Thinking (.45), suggesting
that individuals who quickly process and analyze verbal information also tend to focus on
practical concerns and extract relevant information from verbal cues to make informed
decisions.

Finally, the Verbal Information Processing scale shows a strong positive correlation with HPI
Learning Approach (.32), suggesting that individuals who are adept at processing verbal
information also tend to value education and staying up-to-date with developments in their
field. As a whole, these results suggest the verbal information processing scale assesses an
individuals ability to process detailed information presented in text and use that information
in decision-making.

Numerical Information Processing. Table 3.9 provides selected significant correlations


between scores on the Numerical Information Processing scale and cognitive content from
other assessments.

Table 3.9
Construct Validity Evidence for Numerical Information Processing Scale
Assessment Scale Numerical Informational Processing
SPA Mathematical Ability .20**
BRI Computing Cluster .25**
JPI-R Complexity .22**
HPI Inquisitive .22*
Learning Approach .26**
HBRI Strategic Thinking .41**
Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level.

Across assessments, Numerical Information Processing scale scores show significant


positive correlations with SPA Mathematical Ability (.20), BRI Computing (.25), and JPI-R
Complexity (.22) under the Analytical cluster. These results suggest that individuals who

39

3. VALIDITY

enjoy mathematical computations can also quickly process and analyze numerical data
when making decisions.

Additionally, scores on the Numerical Information Processing scale correlate significantly


with HPI Inquisitive (.22) and Learning Approach (.26), suggesting that individuals who can
quickly recognize patterns in numerical data also tend to be curious and critical thinkers
who prefer to keep up-to-date with developments in their industries.

Finally, the strongest noted correlation for Numerical Information Processing scale scores is
with HBRI Strategic Thinking (.41), suggesting that individuals who possess superior
analytical skills also tend to see the big picture in their analyses of numerical data and
leverage those analyses to make strategic decisions with long-term implications.

3.2 Correlations with Others Descriptions

One of the most important sources of information for interpreting personality scales is
correlations between scale scores and descriptions of a person as rated by observers.

We obtained this information from the Eugene Springfield Community Sample (ESCS;
Goldberg, 2008) using persons who completed the Hogan Judgment Assessment and had
previously been rated by observers on standardized checklists. In the following section, we
provide selected correlations between judgment scales and these observer ratings. For full
correlation matrices with these instruments, please refer to Appendix B.

3.2.1 Procedure

3.2.2 Samples and Instruments

Adjective Descriptions and Personality Phrases. As part of Goldbergs (2008) longitudinal


community research, respondents and observers (e.g., significant others, spouses, friends,
acquaintances, coworkers) completed the Self/Peer Inventories, which includes 88 items
taken from Sauciers (1994) 40-item Big-Five Mini-Markers, the 44-item Big-Five Inventory
(John & Srivastava, 1999; Benet-Martinez & John, 1998), and two additional items in each
inventory measuring physical attractiveness. In this survey, respondents described how well
each adjective or phrase described either themselves or the target individual using a 5-point
Likert scale ranging from 1 (Extremely Inaccurate) to 5 (Extremely Accurate).

Each participant, and up to four observers of each participant, completed these 88 items.
The sample of 196 participants providing self-ratings included 87 males and 109 females.
Ages of subjects ranged from 21 years to 72 years with a mean of 45.45 years (SD = 8.72).
Observers also responded to items assessing how and how well they knew the target, how
much they liked the target, and basic demographic questions on gender and age.

The sample of 538 respondents providing observer ratings included 208 males and 330
females. Ages of observers ranged from 7 years to 89 years with a mean of 41.50 years (SD
= 16.24). Observers were split between spouses and other relatives (N = 300) compared
with friends, coworkers, acquaintances, and significant others (N = 207), with 31 observers

40

3. VALIDITY

not indicating their relationship to the target. Most observers indicated knowing the target
well or very well (N = 522), and most indicated that they liked the target or liked the
target very much (N = 520).

For each of the 88 items, observer ratings were pooled into a composite by calculating a
mean response across all observers. We used these mean responses (N = 196) as the
basis for calculating correlations between observer ratings and the Hogan Judgment Report
scales.

3.2.3 Results of Scale and Observer Description Correlates

The following section presents selected significant correlations with observer ratings that
align with our theory of each scales content. Tables B.17 and B.18 in Appendix B provide
complete correlation matrices for adjective and phrase descriptors, respectively.

Threat Avoidance vs. Reward Seeking. Table 3.10 provides the five strongest negative
correlations and the five strongest positive correlations between Threat Avoidance vs.
Reward Seeking scale scores and observer ratings for descriptive adjectives and phrases.

Table 3.10
Correlations with Threat Avoidance vs. Reward Seeking and Observer Ratings
Descriptor Correlation Descriptor Correlation
Uncreative -.26** Complex .21**
Bashful -.19** Imaginative .19**
Sympathetic -.16* Disorganized .16*
Organized -.14* Philosophical .16*
Efficient -.13 Energetic .15*
Prefers routine work -.32** Full of energy .22**
Likes to cooperate with others -.19** Curious about many different things .22**
Considerate and kind to almost everyone -.17* Has an active imagination .21**
Helpful and unselfish with others -.15* Inventive .16*
Perseveres until the task is finished -.15* Original/comes up with new ideas .15*
Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level; Negative correlations
indicate relationships with Threat Avoidance; Positive correlations indicate relationships with Reward Seeking.

Results suggest that Threat Avoidant individuals seem uncreative (-.26) and preferring
routine work (-.32), whereas Reward Seeking individuals seem complex (.21), energetic
(.22), naturally curious (.22), and imaginative (.21).

Tactical vs. Strategic Thinking. Table 3.11 provides the five strongest negative and positive
correlations between Tactical vs. Strategic Thinking scale scores and observer ratings.

41

3. VALIDITY

Table 3.11
Correlations with Tactical vs. Strategic Thinking and Observer Ratings
Descriptor Correlation Descriptor Correlation
Bashful -.37** Bold .42**
Shy -.30** Extraverted .31**
Quiet -.27** Complex .26**
Withdrawn -.25** Energetic .24**
Unintellectual -.21** Intellectual .24**
Sometimes shy/inhibited -.36** Has an assertive personality .39**
Prefers routine work -.34** Outgoing/sociable .35**
Is reserved -.28** Generates a lot of enthusiasm .29**
Tends to be quiet -.28** Has an active imagination .24**
Tends to be lazy -.23** Ingenious/deep thinker .24**
Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level; Negative correlations
indicate relationships with Tactical Thinking; Positive correlations indicate relationships with Strategic Thinking.

Results suggest that Tactical Thinkers are described as bashful (-.37), inhibited (-.36),
preferring routine work (-.34), and shy (-.30), whereas Strategic Thinkers are described as
bold (.42), assertive (.39), outgoing (.35), and extraverted (.31).

Data-Driven vs. Intuitive Decisions. Table 3.12 provides the five strongest negative and
positive correlations between Data-Driven vs. Intuitive Decisions scale scores and observer
ratings.

Table 3.12
Correlations with Data-Driven vs. Intuitive Decisions and Observer Ratings
Descriptor Correlation Descriptor Correlation
Complex -.31** Good-Looking .20**
Quiet -.30** Talkative .17*
Practical -.18* Unintellectual .17*
Intellectual -.18* Extraverted .14*
Systematic -.14* Bold .14*
Tends to be quiet -.28** Physically attractive .22**
Curious about many different things -.23** Outgoing/sociable .16*
Likes to reflect/play with ideas -.22** Generates a lot of enthusiasm .13
Ingenious/deep thinker -.21** Gets nervous easily .13
Is reserved -.19** Has an assertive personality .12
Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level; Negative correlations
indicate relationships with Data-Driven Decisions; Positive correlations indicate relationships with Intuitive
Decisions.

Results suggest that individuals who base their decisions on data seem complex (-.31) and
quiet (-.30), whereas intuitive decision-makers seem physically attractive (.22), talkative
(.17), and unintellectual (.17).

Defensive vs. Cool-Headed. Table 3.13 provides the five strongest negative and positive
correlations between Defensive vs. Cool-Headed scale scores and observer ratings.

42

3. VALIDITY

Table 3.13
Correlations with Defensive vs. Cool-Headed Reactions and Observer Ratings
Descriptor Correlation Descriptor Correlation
Temperamental -.30** Relaxed .30**
Moody -.26** Cooperative .22**
Jealous -.24** Kind .19**
Harsh -.23** Unenvious .19**
Touchy -.23** Practical .11
Can be tense -.32** Emotionally stable/not easily upset .38**
Finds fault with others -.28** Relaxed/handles stress well .33**
Is depressed/blue -.28** Remains calm in tense situations .24**
Gets nervous easily -.28** Makes plans and follows through .21**
Worries a lot -.27** Likes to cooperate with others .19**
Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level; Negative correlations
indicate relationships with Defensive reactions; Positive correlations indicate relationships with Cool-Headed
reactions.

Results suggest that Defensive individuals seem tense (-.32) and temperamental (-.30),
whereas Cool-Headed individuals seem not easily upset (.38), adept at handling stress (.33),
and relaxed (.30).

Denial vs. Acceptance. Table 3.14 provides the five strongest negative and positive
correlations between Denial vs. Acceptance scale scores and observer ratings.

Table 3.14
Correlations with Denial vs. Acceptance and Observer Ratings
Descriptor Correlation Descriptor Correlation
Bold -.29** Bashful .24**
Complex -.28** Uncreative .19**
Intellectual -.25** Unintellectual .18*
Energetic -.20** Shy .15*
Extraverted -.20** Withdrawn .12
Ingenious/deep thinker -.28** Prefers routine work .29**
Has an assertive personality -.27** Sometimes shy/inhibited .24**
Original/comes up with new ideas -.25** Tends to be lazy .21**
Inventive -.23** Is reserved .15*
Full of energy -.22** Has a forgiving nature .14*
Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level; Negative correlations
indicate relationships with Denial; Positive correlations indicate relationships with Acceptance.

Results suggest that people who deny the validity of feedback seem bold (-.29), complex (-
.28), and assertive (-.27), whereas individuals who accept feedback seem to prefer routine
work (.29), and may be bashful (.24) and sometimes inhibited (.24).

Superficial vs. Genuine Engagement. Table 3.15 provides the five strongest negative and
positive correlations between Superficial vs. Genuine Engagement scale scores and
observer ratings.

43

3. VALIDITY

Table 3.15
Correlations with Superficial vs. Genuine Engagement and Observer Ratings
Descriptor Correlation Descriptor Correlation
Withdrawn -.28** Bold .28**
Bashful -.26** Energetic .18*
Fretful -.21** Practical .15*
Shy -.21** Extraverted .13
Moody -.15* Complex .12
Gets nervous easily -.28** Outgoing/sociable .24**
Is depressed/blue -.26** Has an assertive personality .23**
Sometimes shy/inhibited -.21** Full of energy .22**
Is reserved -.19** Relaxed/handles stress well .20**
Can be moody -.18* Remains calm in tense situations .19**
Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level; Negative correlations
indicate relationships with Superficial Engagement; Positive correlations indicate relationships with Genuine
Engagement.

Results suggest that Superficially Engaged individuals seem nervous (-.28), withdrawn (-.28),
and bashful (-.26), whereas Genuinely Engaged individuals seem bold (.28), outgoing (.24),
assertive (.23), energetic (.22), and adept at handling stress (.20).

Openness to Feedback and Coaching. Table 3.16 provides the five strongest negative and
positive correlations between Openness to Feedback and Coaching scale scores and
observer ratings.

Table 3.16
Correlations with Openness to Feedback & Coaching and Observer Ratings
Descriptor Correlation Descriptor Correlation
Temperamental -.30** Relaxed .25**
Jealous -.25** Cooperative .21**
Harsh -.25** Unenvious .14
Fretful -.24** Practical .13
Touchy -.24** Warm .13
Is depressed/blue -.32** Emotionally stable/not easily upset .33**
Gets nervous easily -.30** Relaxed/handles stress well .23**
Finds fault with others -.26** Remains calm in tense situations .22**
Worries a lot -.25** Has a forgiving nature .19**
Can be tense -.25** Likes to cooperate with others .18**
Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level.

Results suggest that individuals who resist coaching seem depressed (-.32), temperamental
(-.30), and nervous (-.30), whereas individuals who are generally receptive to feedback and
coaching seem emotionally stable (.33), relaxed (.25), adept at handling stress (.23), and
calm in tense situations (.22).

Verbal Information Processing. Table 3.17 provides the five strongest negative and positive
correlations between Verbal Information Processing scale scores and observer ratings.

44

3. VALIDITY

Table 3.17
Correlations with Verbal Information Processing and Observer Ratings
Descriptor Correlation Descriptor Correlation
Unintellectual -.23** Organized .21**
Disorganized -.22** Efficient .17*
Talkative -.19** Deep .17*
Temperamental -.13 Intellectual .17*
Touchy -.11 Systematic .15*
Easily distracted -.18* Does things efficiently .20**
Tends to be disorganized -.17* Original/comes up with new ideas .15*
Can be somewhat careless -.16* Ingenious/deep thinker .14
Outgoing/sociable -.12 Does a thorough job .13
Starts quarrels with others -.10 Sophisticated in art, music, literature .12
Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level.

As seen in Table 3.17, correlations between Verbal Information Processing scale scores and
observer ratings suggest that individuals who struggle to process verbal information seem
unintellectual (-.23), disorganized (-.22), and talkative (-.19), whereas those who can quickly
process this information seem organized (.21), efficient (.20), and intellectual (.17).

Numerical Information Processing. Table 3.18 provides the five strongest negative and
positive correlations between Numerical Information Processing scale scores and observer
ratings.

Table 3.18
Correlations with Numerical Information Processing and Observer Ratings
Descriptor Correlation Descriptor Correlation
Unintellectual -.19** Intellectual .22**
Talkative -.12 Practical .16*
Uncreative -.12 Deep .13
Jealous -.11 Systematic .13
Fretful -.11 Unenvious .11
Tends to be disorganized -.10 Ingenious/deep thinker .18**
Is depressed/blue -.08 Sophisticated in art, music, literature .15*
Gets nervous easily -.08 Makes plans and follows through .15*
Easily distracted -.07 Likes to reflect/play with ideas .13
Has few artistic interests -.07 Original/comes up with new ideas .12
Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level.

Results suggest that individuals who struggle to process quantitative data seem
unintellectual (-.19) and uncreative (-.12), whereas those who can quickly process numerical
information seem intellectual (.22) and deep thinking (.18).

Taken together, these results provide additional support for the construct validity of the
Hogan Judgment Report scales. More importantly, these results indicate that an individuals
natural decision-making tendencies, reactions to feedback, receiving feedback and
coaching, and processing verbal and numerical information are reflected in that persons
reputation (i.e., how others describe them).

45

3. VALIDITY

3.3 Criterion-Related Validity

Aguinis, Henle, and Ostroff (2001) described criterion-related validity in terms of the
relationship between a predictor and some criterion measure (e.g., job performance), with
the goal of answering the basic question, How accurate are the assessment scores in
predicting criterion performance? The Uniform Guidelines state that evidence of the
validity of a testby a criterion-related validity study should consist of empirical data
demonstrating that the [test] is predictive of or significantly correlated with important
elements of job performance (29 C.F.R. 1607.5 (B)).

Although many organizational and logistical constraints limit the usefulness of criterion-
related validity studies (McPhail, 2007), the Uniform Guidelines and Principles suggest
considering this approach when a) there is an adequate, representative sample of job
incumbents willing to participate, and b) development of reliable, unbiased measures of job
performance is possible. The Principles also recommends using a relevant criterion
measure, one that reflects the relative standing of employees with respect to important
work behavior(s) or outcome measures(s) (p. 14).

3.3.1 Procedure

We are collecting criterion-related validation data for the Hogan Judgment Report scales as
part of global research studies, using online, unproctored Internet testing. Specifically, we
have offered complimentary Hogan Judgment Reports to individuals in professional and
managerial roles across the globe in exchange for their completion of the Hogan Judgment
Assessment and their supervisors completion of a conceptually-aligned performance rating
form described below. No participants took the assessment as part of high stakes testing
where results impacted hiring, promotion, or other personnel decisions.

3.3.2 Sample and Instrument

Judgment-Related Performance Rating Form. To collect conceptually-aligned job


performance criterion data, we created a judgment-related Performance Rating Form (PRF)
based on our theory of each scales content. The PRF includes 34 questions and is divided
into four sections: (a) overall performance, (b) decision-making descriptions, (c) cognitive
ability, and (d) open-ended comments.

For the first section, we ask supervisors to rate (a) the employees overall job performance
using a scale ranging from 1 (Not Effective) to 5 (Exceptional), (b) whether the employee is a
model for new employees work behavior using a scale ranging from 1 (Strongly Disagree) to
5 (Strongly Agree), (c) what their response would be if the employee was thinking of leaving
the organization, with options ranging from 1 (Do Nothing) to 3 (Make a Significant Effort to
Keep Him/Her), and (d) how often they agree with the employees decisions, using a scale
ranging from 1 (Rarely) to 4 (Always). Remaining questions in this section assess whether
the employee (a) exemplifies organizational values, (b) makes good decisions at work, and
(c) usually seeks out coaching or feedback on performance. Rating scales for these
remaining questions use the Strongly Disagree Strongly Agree scale previously described.

46

3. VALIDITY

To examine decision-making, we present 10 pairs of descriptors (e.g., This employee tends


to focus on the details vs. This employee tends to focus on the big picture) and ask
supervisors to (a) decide which descriptor best describes the employee, and (b) whether the
employee shows either a Moderate Preference or Strong Preference for the descriptor that
accurately describes them.

Next, we ask supervisors to rate the employee on 10 items related to general cognitive
ability in the context of making work-related decisions. Example items ask whether the
employee gathers essential information needed to make decisions, seeks out and considers
alternative decisions, makes good decisions under ambiguous circumstances, and makes
good decisions under stress. Supervisors respond to these items using a scale ranging from
1 (Strongly Disagree) to 5 (Strongly Agree).

Finally, we provide supervisors with an opportunity to provide open-ended comments about


their employees work-related decision-making. Specifically, we ask supervisors to describe
(a) the employees three biggest strengths related to decision-making, and (b) their three
biggest challenges to making work-related decisions.

At present, we have collected data from approximately one hundred individuals who have
completed the Hogan Judgment Assessment and been rated by their supervisors on the
Performance Rating Form. Although this sample would allow us to report favorable
preliminary findings, our current sample size does not allow us to draw generalizable
conclusions based on statistically stable results. As such, we will report these findings as
soon as we have obtained data from a sufficiently large and representative sample.

47

4. INTERPRETATION & USES

4. Interpretation and Uses

The Hogan Judgment Report is designed primarily for leadership development, training and
coaching focused on decision-making. We assume that making people aware of their typical
decision-making tendencies should help them make better future decisions. The Judgment
Report is suitable for pre-hire assessment when the goal is to ensure that a candidates
decision-making and thinking styles are a good fit with the demands of a role, team, or
organization.

The Judgment Report provides feedback and developmental tips in four areas: (a) numerical
and verbal information processing, (b) natural pre-decision tendencies, (c) reactions to
feedback about failed decisions, and (d) overall coachability. The report can help create
strategic self-awareness concerning decision-making and responses to feedback regarding
bad decisions, both of which are critical to job performance, career success, and leadership
effectiveness.

Below we describe characteristics associated with different scores for each of the Judgment
scales. These interpretive statements were derived from (a) descriptions provided by co-
workers, and (b) demonstrated relationships between the scales and respondents scores
on well-established personality, values, and ability measures. We also provide some
developmental suggestions; these recommendations must be tailored to the specific needs
of the individual and will be most effective when implemented in consultation with a trusted
mentor or coach.

4.1 Information Processing

People differ in speed of processing information, and some are faster than others. These
differences are usually attributed to short-term memory capacity and attentional focus.
Information processing capacity enables people to solve well-structured problems by
identifying patterns, making connections, or thinking abstractly.

There are two main domains of information processing verbal and numerical. Verbal tests
concern the ability to solve problems based on words or concepts. Numerical tests concern
a persons ability to solve problems based on numbers or abstract reasoning. The
information processing section of the Hogan Judgment Report concerns both verbal and
numerical information processing. Below, we provide interpretive content for scores in each
of these sections.

4.1.1 Verbal Information Processing

The verbal section evaluates the ability to process information presented with words or
verbal content. Although the problems in this section require little previous knowledge or
vocabulary, people who perform well should have stronger verbal ability. They tend to be
interested in the humanities and are generally a good fit for careers that require using words
to convey emotions and tell storiese.g., literature, journalism, law, and education.
Conversely, individuals with lower scores on this section typically require more time to
process verbal information.

48

4. INTERPRETATION & USES

4.1.2 Numerical Information Processing

The numerical information processing section evaluates the ability to process information
presented only in numerical data. Although the problems in this section require only basic
command of arithmetic, people who perform well should have stronger math skills and good
logical reasoning ability. They tend to be interested in scientific fields and are generally a
good fit for careers that require data-based problem solvinge.g., mathematics, engineering,
accounting, and Information Technology. Conversely, individuals with lower scores on this
section typically require more time to process numerical data.

4.1.3 Information Processing Styles

As shown in Figure 4.1, people tend to fall into one of four groups depending on their
information processing style. The distinction between qualitative and quantitative
information processing styles is based on C.P. Snows (1959) notion of two cultures,
namely Sciences vs. Humanities, and highlights the two different modes of information
processing used in decision-making. We present interpretive and developmental content for
each information processing style below.

Figure 4.1
Information Processing Styles

4.1.3.1 Deliberate

Individuals with average scores for both verbal and numerical information processing tend
to analyze information in a deliberate manner. They take their time in making decisions to
ensure that they understand the issues completely. They tend to be described by others as
organized and systematic, and are often viewed as practical and detailed.

49

4. INTERPRETATION & USES

4.1.3.2 Quantitative

Individuals with high scores for numerical, but average scores for verbal information
processing enjoy analyzing mathematical problems and prefer to focus on facts rather than
people. They are often good at translating data into guidelines for decision-making, and
view life as a series of problems to be solved. Quantitative information processors often
excel at evaluating factual issues, and prefer concrete challenges to ambiguously defined
problems. They tend to be described as ingenious deep-thinkers, and are often viewed as
sophisticated and good at following through.

4.1.3.3 Qualitative

Persons with high scores for verbal, but average scores for numerical data processing, tend
to be good at using language to analyze problems, communicate decisions, and express
their emotions. This information processing style also facilitates understanding and
interpreting others feelings and intentions. Qualitative information processors are more
interested in people issues than abstract, logical, or mathematical problems. Others tend to
describe them as deep, philosophical, and original thinkers who often generate new ideas.

4.1.3.4 Versatile

Individuals with high scores for both numerical and verbal information processing are
versatile information processors who can effectively use both numerical and verbal
information for decision-making. They tend to do well in many fields. In general, they are
insightful about both people issues and data-based problems. Others often describe them
as intellectual thinkers who can process information and follow through on decisions.

4.2 Decision-Making Approaches

Another key aspect of individual differences in judgment concerns how people approach
decisions. Unlike information processing style, decision-making tendencies involve
automatic, spontaneous, and subconscious tendencies, or dispositional strategies that
people adopt when solving ill-defined or ambiguous problems. Because most of the
problems we encounter at work dont have an objectively correct solution and contain
insufficient information to demand careful analysis, we normally rely on mental shortcuts
spontaneous decision heuristics that represent our default problem solving tendencies.

People make business decisions using certain predictable tendencies: (a) threat avoidance
vs. reward seeking; (b) tactical vs. strategic thinking; and (c) data-driven vs. intuitive
decisions. Every individual can be described in terms of their placement along these three
dimensions. For example, people with strong reward seeking tendencies are likely to
approach decisions with an eye on the prize they tend to evaluate options based on
potential positive outcomes and select the approach that will lead to the biggest reward. On
the other hand, people who fear failure tend to evaluate options based on potential
negative consequences and avoid risky choices with the potential for bigger rewards.
People with a more balanced approach consider both potential rewards and risks
associated with each possible course of action.

50

4. INTERPRETATION & USES

The Hogan Judgment Report describes the extent to which a person relies on the
tendencies defined by these scale pairs. Understanding these tendencies can:
Enhance an individuals strategic self-awareness of his/her decision-making style
Help people identify resources to complement their natural decision-making style
Facilitate professional development by highlighting potential strengths and challenges
inherent in a persons natural decision-making tendencies
Inform talent decisions (e.g., selection and placement) by identifying individuals whose
natural decision-making tendencies fit the requirements of a role
Facilitate coaching and training efforts by highlighting potential challenges that may arise
as the individual attempts to change and improve

4.2.1 Threat Avoidance vs. Reward Seeking

People who are threat avoidant focus on the negative side of the risk-reward equation; they
make decisions intended to minimize potential financial, legal, physical, material, personnel,
or other losses. Conversely, people who are reward-seeking focus on the positive side of the
risk-reward equation; they make decisions intended to maximize their potential rewards. In
simple terms, threat-avoidance concerns being motivated by avoiding the sticks, whereas
reward-seeking concerns being motivated by earning the carrots.

Threat avoidant individuals try to anticipate future threats and potential risks. When
evaluating options during decision-making, they focus on what could go wrong. They
approach decision-making opportunities in a structured manner, and often follow a defined
process to evaluate the situation and potential courses of action. As a result, others may
see them as predictable and cautious in their problem-solving and decision-making.
Although colleagues may see them as reliable and dependable, threat avoidant decision-
makers may not solicit input in the decision-making process, because other viewpoints may
be distracting. Threat avoidant people also struggle to influence others to support their
decisions, and may seem overly concerned about unlikely or relatively insignificant problems.
In many cases, they may choose the alternatives that include the fewest unknowns and are
likely to result in positive outcomes even if the benefits are modest. Although others may
not expect them to make radical or exciting decisions, they may be relied upon consistently
to make solid, safe decisions.

In contrast, reward seeking individuals stay alert for opportunities for gain and growth.
When making decisions, reward seekers may emphasize the potential positive outcomes
associated with various options. They tend to adopt a big risk, big reward attitude and
seek to maximize the possible gains associated with every decision. They regard each
decision point as a potential big win. As a result, they may choose alternatives with great
potential even if the alternatives are obviously dangerous. Reward seekers often take a
dynamic approach, using a process that can be adapted to explore any potentially
advantageous paths discovered during decision-making. Reward seekers may also include
others, leveraging connections to build support for decisions, including those that involve
significant risk. They are often able to influence others to move outside their comfort zones
and embrace risks in order to achieve rewards. Nonetheless, reward seekers may take
unnecessary or even inadvisable risks to pursue potential rewards. They may ignore

51

4. INTERPRETATION & USES

processes and rules that impede their ability to make what they consider to be optimal
decisions. Others tend to see them as interesting and engaging, but also prone to testing
limits and taking risks without fully considering the consequences of their actions. Reward
seekers may be impulsive and take action quickly. Although this may sometimes pay off, it
will also lead to avoidable losses along with wins. Consequently, reward seekers may
benefit from soliciting others input to see if they perceive the same opportunities. Reward
seekers can benefit from considering potential risks as well as rewards.

4.2.1.1 Developmental Recommendations for Threat Avoidance

During decision-making, ask others what risks they perceive in the situation. Do they see
the same threats and potential negative side effects in each alternative? If so, how
concerned are they about the threats?
Frame your decision-making rationale in optimistic terms by starting with positives, then
discussing potential hazards, and then finishing on a positive note.
When next faced with an opportunity that seems too risky, consider the worst that might
happen if you did take action. If the worst case scenario is relatively innocuous, you
should practice taking risks and moving to action despite uncertainty.
Create a process for evaluating risk, so that you have a consistent method for analyzing
the pros and cons of situations, determining the most appropriate action to take, and
evaluating the outcomes of that action.
Identify past situations in which you took risks and they paid off. What situational factors
help you feel comfortable (e.g., working collaboratively, working on lower stakes projects,
working with a leader who supports risk taking)? Consider how you might replicate those
factors in future situations.

4.2.1.2 Developmental Recommendations for Reward Seeking

Before making decisions, critically review your alternatives in terms of the amount of risk
involved. Identify options that can reduce potential unintended consequences
associated with the decision. Small changes may reduce unnecessary risk or help you
better define the unknowns in the situation.
Use your team to help you identify the risks in a given decision. Your more cautious
colleagues may see things that you don't, and their insights may help you better
understand what could go wrong and how others may perceive your decision once its
made.
Identify situations in which you may have taken a "ready, fire, aim" approach. What
factors lead you to move quickly without fully evaluating the circumstances? Do you
tend to act quickly when you are bored? When you are under heavy pressure? When you
are in routine situations? Identify checks-and-balances that you can use in these
situations to ensure you properly consider your options before moving to action.
For the next few months, keep a journal of your significant decisions. List both the pros
and cons of your decision and ask your boss or a trusted associate to do the same.
Once you have created a record, review each decision. How many times did you "win"?
What risks or potential negative consequences did you fail to consider prior to making a

52

4. INTERPRETATION & USES

decision? How could your decision-making process be improved to increase the number
of wins?
There is a difference between seeking input from others and seeking their buy-in. Before
making decisions, seek constructive feedback on your approach. Find ways to
incorporate that feedback into your decision-making process. It may not change your
final decision, but you can at least use it to improve your approach, inform how you
manage the potential negative impact of the decision, or guide how you evaluate the
decision after its made. Input from others may improve your process and decisions and
demonstrate that you value their opinions and are not just trying to influence them to
support you.

4.2.2 Tactical vs. Strategic Thinking

Tactical thinkers approach decisions with a short-term, detail-oriented perspective. They


focus on immediate concerns such as cost, implementation issues, and staffing problems
and tend not to put these issues in a larger context. Conversely, strategic thinkers tend to
put problems in context, and look for systematic or comprehensive solutions to problems
rather than treating symptoms as they occur; they often initiate strategic change but may
ignore more detailed issues such as cost and implementation.

Tactical thinkers are skilled at identifying troublesome details and implementing programs.
They approach decisions by identifying the specific goal that the decision is meant to
achieve, then finding the most direct route to that outcome. Put another way, tactical
thinkers focus on the journey rather than the destination. They often seem to prefer to let
others define the goals rather than propose their own vision or strategy. Tactical thinkers
tend to work backwards from the desired goal to identify the concrete steps needed to
achieve that goal and mitigate the risk of overlooking critical details. Others see them as
detail-oriented and methodical, although they may sometimes get lost in the details and lose
sight of the broader implications of the decision. Tactical thinkers often rely on established
practices and proven solutions to guide their decisions; this may lead others to view them as
efficient decision-makers because they dont spend time considering untenable approaches
that have to eventually be abandoned. However, tactical thinking may lead to missed
opportunities for more innovative or forward-thinking decisions. Tactical thinkers are usually
well-equipped to make pragmatic decisions that require synthesizing detailed information.
Additionally, they work well with more innovative problem-solvers, providing checks and
balances to ensure that important operational factors are not overlooked in the decision-
making process.

Strategic thinkers tend to focus on identifying larger business trends and developing
overarching business plans. When making decisions, they pursue broad, flexible objectives,
and avoid defining concrete, narrow goals. They are often reluctant to choose specific paths
to avoid closing the door on both known and unknown possibilities. They often rely on an
overall strategy or vision to guide their decision-making, focusing on the destination more
than the journey. As a result, others may view strategic thinkers as adaptable and able to
follow multiple paths to make the best decision in a given situation. Strategic thinkers tend
not to rely on standard processes or methodologies, preferring instead to let specific
situations dictate their decision-making approach. During the decision-making process, they

53

4. INTERPRETATION & USES

may continuously respond to new information and changing circumstances and adapt their
thinking to the changing decision landscape. Consequently, others may find it difficult to
predict the strategic thinkers decisions. Others may believe in a strategic thinkers ability to
set strategy and articulate a vision, but question his/her ability to consider the details
entailed by their strategies. Strategic thinkers shine when they can develop a 30,000-foot
view strategy. But when faced with routine decisions, they may become bored and take
action without considering important nuances of the situation that should inform the
decision. They may work well with tactical decision-makers who focus on the operational
and logistical details that strategic thinkers ignore.

4.2.2.1 Developmental Recommendations for Tactical Thinkers

Focus on the global and macro-economic trends in your industry and connect them with
your organizations strategy. Identify the ways in which the decisions you make relate to
(a) your organizations broader strategy, and (b) the conditions in your industry.
Think of a time when you found it difficult to make an important decision in an
ambiguous situation. What were the specific unknowns in the situation? How did you
address these factors in order to make a decision? Consider your typical decision-
making process and how you might adapt it to better manage unexpected challenges
when they arise in the future. Identify techniques you might use to manage uncertainty
so that it doesnt impede your decision-making. For example, you might ask others to
help you better define the unknowns in a given situation, or develop contingency plans
so that your decision can be modified if new information comes to light.
You may approach decisions by first defining the goal. Consider how the immediate goal
might serve a larger mission or vision. Try to document how your immediate goal might
contribute to the team or organizations long-term objectives. Share this with a trusted
colleague, supervisor, or mentor and ask them to fill in other connections and paths that
might not have occurred to you.
Look for ways your decisions might conflict with broader strategies and objectives.
Consider how your present choices could conflict with some desired future state. Use
this forward-thinking perspective to inform your decisions by both their short-term,
known consequences and their potential future consequences. Try to balance these
critical perspectives when making decisions.
Identify key stake-holders who may take a more strategic approach to decision-making.
Include them in your decision-making process, so that you benefit from their big-picture
perspective while incorporating your own strengths around detail-orientation and day-to-
day operational decision-making.

4.2.2.2 Developmental Recommendations for Strategic Thinkers

When making decisions you may prefer loosely defined situations and goals as they may
allow for a greater degree of freedom in decision-making. The next time you are faced
with a decision, document the known information that will impact your decision as well
as the related unknowns. Evaluate these unknowns and identify areas that need better
definition. Ask others to help you, so that you dont move too quickly towards a decision
without needed information.

54

4. INTERPRETATION & USES

Identify team members who are more detail-oriented and tactical than you are. Include
these people in your decision-making so that they can help you better understand the
cost and implementation issues involved in the decision.
When considering big-picture strategies, ask a team member to play devils advocate.
Rely on that individual to challenge your decisions. Ask this person to question your
approach when he/she feels critical considerations are being glossed over. Doing this
may help you avoid ineffective decisions that will eventually have to be abandoned. In
addition, this persons input may improve your decision-making and positively impact the
immediate situation and future decisions you will make.
After articulating a strategy, work backward to define small tactical wins that may
contribute to the overall objective. Posting small wins may help others buy into your
strategy.
Consider adding more structure to your decision-making process. You only need a few
replicable techniques that you can use in any decision situation (e.g., setting
intermediate goals, considering main and side effects in a systematic manner,
introducing regular check points as you purse overall objectives). Increasing the
predictability of your decision-making process will allow others to assist you in making
effective decisions and mitigate the risk their being surprised or confused by your
decisions.

4.2.3 Data-Driven vs. Intuitive Decisions

Data-driven individuals tend to carefully review the relevant data and facts before making
decisions. These individuals may also revisit and revise their decisions periodically to
realign them with new data. Conversely, intuitive people rely more on their experience and
gut instincts, preferring to make quick decisions and move on.

Data-driven decision makers prefer to review large amounts of information and use it to
make sound decisions. They seem to approach decisions as puzzles to be solved. They
typically gather information from sources immediately related to the situation as well as
those further from the decision at hand; as a result, they may become overwhelmed by the
data and struggle to prioritize the facts in terms of their importance to the decision. After
gathering all available information, they analyze it from many different angles, incorporating
multiple perspectives and considering the individual pieces of data and the sum of the
information. They value well-supported decisions over timely decisions. They may let the
results of their analysis drive their decisions, even if they defy intuition, past experience, or
conventional wisdom. As such, they may be most comfortable and effective in situations
where there is a finite amount of information related to the decision, and those facts provide
a strong foundation for the ultimate decision. However, they may struggle when there is
insufficient information or when the facts do not point to the best answer.

In contrast, intuitive decision-makers prefer to base their decisions on past experience and
gut instincts. They may be comfortable acting without thoroughly analyzing the factors at
play in each situation. They approach decision-making with confidence that their innate
talent, previous experience, common sense, and business savvy will lead them to effective
decisions. They tend to be comfortable choosing among possible approaches even if they
are not well-informed about all possible approaches. They often take a fluid approach to

55

4. INTERPRETATION & USES

decisions, feeling that meticulous analysis may be unnecessary at best and detrimental at
worst. They may make creative decisions because they dont need evidence to support
decision alternatives. Intuitive decision-makers may be bored by reviewing seemingly trivial
information and prefer to quickly identify the decision to be made, then select and
implement an approach. Such people may be admired for their confidence, but may also be
seen as flying by the seat of their pants. They may perform well in situations where timely
decisions are needed and clear evidence or supporting information is unavailable. However,
at other times they may fail to consider important data and facts in their decision-making
process.

4.2.3.1 Developmental Recommendations for Data-Driven Decisions

When analyzing data needed to make a decision, consider how various organizational
perspectives may affect your interpretation of the data.
Think of the last time you made a bad decision despite having thoroughly analyzed the
available evidence. Try to identify warning signs that may have preceded the decision.
Did others on your team, whom you could include in your future decisions, doubt the
decision?
Consider a time when you were overwhelmed by the information you had gathered in
order to make a decision. How might you have better prioritized the information? Identify
techniques you can use to better manage decision-related data in future situations, such
as categorizing evidence in terms of shared content, rating each piece of data as first,
second, or third tier in terms of value and importance, and discarding information that
may be only superficially related to the decision.
When proposing solutions to complex problems, consider your audience. Although you
may value data and analytics, others may prefer an executive summary. Highlight your
critical findings and provide others with key take-aways related to your decision; this
may be more persuasive than providing a detailed description of all of the factors
contributing to your decision.
When approaching a new decision point, set a deadline for the decision and timelines for
each activity needed for the final decision. Creating a decision-making schedule may
mitigate the risk of delaying critical decisions.

4.2.3.2 Developmental Recommendations for Intuitive Decisions

Before making a decision, document your reasoning in case you are asked to provide
more information on your approach. Delegate fact gathering and data analysis to team
members who may be motivated by opportunities to engage in these investigative
activities.
Listen carefully when others present data that conflict with your past experience or
intuition. You will show that you value their input and expertise, and heeding others
warnings may prevent you from becoming overly confident based on hunches that paid
off in the past. Considering others concerns about your decisions may help you
challenge your own thinking, which may lead to higher-quality decisions.
Periodically re-evaluate your decisions, noting what worked and what you might have
done differently had you known then what you know now. Look specifically for decisions

56

4. INTERPRETATION & USES

where you missed information that might have prevented you from going down an
unproductive path and having to change course afterwards.
Consider the impact your decisions may have on your team and the wider organization.
Identify the kinds of decisions you are required to make for which you may need to rely
on more than your judgment and experience alone. Develop techniques to gather
needed information and evaluate critical facts for those situations.
When making a decision, think about previous experiences that may influence your
approach to the situation. Reflect on the elements that were successful as well as those
that could have been improved. This may mitigate the risk of repeating mistakes and/or
missing the same critical considerations in multiple situations.

4.3 Decision-Making Styles

People differ in their tendencies to (a) avoid threats or seek rewards, (b) think tactically or
strategically, and (c) use data or intuition when making decisions. Each persons decision-
making style reflects a combination of these tendencies. For example, one person may
think strategically and use data to drive decisions to avoid threat. A different person may
seek rewards by focusing on tactical details and following his/her intuitions. The
combinations of tendencies across these scales define eight decision-making styles. These
styles provide a more holistic account of each persons typical decision-making. Figure 4.2
presents the eight decision-making styles, which we describe below. Each decision-making
style is defined by an archetypal occupation that exemplifies the decision-making
characterized by the scores on the three scale pairs. These archetypal occupations are only
intended to describe typical thinking patterns in decision-making, not likely vocational
interests, preferences, or performance.

Figure 4.2
Decision-Making Styles

4.3.1 Auditors

Some individuals are characterized as threat avoidant, tactical thinking, and data-driven
decision-makers. This pattern of decision-making is best represented by Auditors because
such people typically make tactical decisions designed to minimize short-term risks, pay

57

4. INTERPRETATION & USES

attention to details, get involved with day-to-day operations, and make data-based decisions
that they periodically review and adjust.

4.3.2 Surgeons

Individuals characterized as threat avoidant, tactical thinking, and intuitive decision-makers


are labeled as Surgeons. They typically avoid taking unnecessary risks, focus on solving
immediate problems, pay attention to details, base their decisions on practical experience,
and prefer hands-on involvement in the work.

4.3.3 Stock Traders

Some individuals seek tactical, short term rewards using data-driven decisions. Stock
Traders are prototypical of such decision-making tendencies because they focus on short-
term wins, making quick and calculated tactical decisions based in data, and regularly
evaluating the effectiveness of their decision-making processes.

4.3.4 Defense Analysts

Defense Analysts make data driven decisions to defend comprehensively against a broad
range of long term threats. They are focused on strategies that systematically eliminate
threats; they base their decisions on a thorough review of the data, and periodically evaluate
their decision-making methods.

4.3.5 Politicians

Some individuals make quick and intuitive decisions, guided by a long term strategy,
designed to maximize their pay-offs. This pattern of decision-making is best represented by
Politicians because these individuals tend to remain alert for opportunities to advance
their strategic agendas, make decisions to maximize long-term interests, base decisions on
instinct, and generally ignore details such as implementation issues.

4.3.6 Chess Players

Chess Players are people who avoid long term strategic threats by making rapid decisions
based on experience and intuition. We chose this prototype because such people make
decisions designed to minimize threats to future positions, base decisions on strategy, think
several moves ahead, make decisions based on their past experience and intuition, accept
short-term losses to make long-term gains, and expect that success will take time.

4.3.7 Promoters

Some individuals are characterized as reward seeking, tactical thinking, and intuitive
decision-makers. This pattern of decision-making is best represented by Promoters
because such people tend to make rapid decisions designed to maximize immediate
rewards, expect quick results, make decisions based on their experience and intuition,

58

4. INTERPRETATION & USES

spend little time worrying about past mistakes, and show little interest in detailed data
analyses.

4.3.8 Investors

Finally, Investors are people who seek sustainable, long term rewards by making careful,
data-based decisions. We chose this prototype because such people focus on achieving
long-term strategic goals, stay alert for hidden but consistent business trends, accept short-
term losses to earn long-term gains, make data-driven decisions, periodically revisit their
decisions, and try to maximize their potential rewards.

4.4 Reactions to Feedback

Because the base rate for bad business decisions is at least 50% (DeVries, 1992), a crucial
element of business judgment concerns how people react to feedback that their decisions
didnt work out. People tend to respond to negative feedback in one of three typical ways:
(a) by being defensive or remaining cool-headed; (b) by denying or accepting the feedback;
and (c) with superficial agreement or genuine engagement in the feedback. Every individual
can be described in terms of these three dimensions. More importantly, unlike the pre-
decision approaches which have unique strengths at both the low and high ends of each
scale, reaction to feedback scales include undesirable behaviors at the low end (i.e.,
defensiveness, denial, superficial agreement) and desirable behaviors at the high end (i.e.,
remaining cool-headed, accepting feedback, genuinely engaging in feedback).

The Hogan Judgment Report describes the degree to which people typically rely on these
three tendencies when responding to feedback about failed decisions. Understanding
these tendencies can enhance an individuals strategic self-awareness. This information
has implications for the selection, placement, and development of organizational leaders.
Knowing how an individual typically reacts to negative feedback also provides a realistic
preview of their responses to developmental initiatives such as coaching.

4.4.1 Defensive vs. Cool-Headed

Defensive individuals tend to respond to negative feedback by becoming emotional and


projecting blame outwards to people, circumstances, timing, and other factors outside their
control. Such individuals may need time to compose themselves before responding to
negative feedback, and need to find ways to suppress their inclination to react emotionally.

Conversely, cool-headed individuals tend to remain calm and consider how they may have
contributed to incorrect business decisions. Although this response is preferable to
defensiveness, cool-headed individuals may sometimes benefit from demonstrating more
passion when defending their decisions, especially when the facts are on their side. They
might also consider displaying more concern about the negative outcomes to which they
contributed. Otherwise, they may seem unconcerned about the news that their decision did
not work out, and their long-term commitment to their job may be questioned based on that
perceived lack of passion.

59

4. INTERPRETATION & USES

4.4.2 Denial vs. Acceptance

People who deny negative feedback tend to refuse to recognize facts, deflect responsibility,
reinterpret failure as success, and deny that problems exist, or try to move on. They could
benefit from facing criticism and attending to feedback in the future. They could also reflect
on how others may interpret their denial as an inability to learn from experience. They can
use such events as opportunities to learn and improve future decisions.

Conversely, individuals who can accept negative feedback tend to carefully consider the
facts, address failure directly, and leverage negative feedback to improve future decisions.
Although this response is preferable to denying negative feedback, accepting individuals
may also benefit from using feedback to learn about what they did right, not just where they
made mistakes. In addition, these individuals may be too accepting of negative feedback at
times. As such, they should remember that taking responsibility for factors beyond their
control is just as counterproductive as not taking responsibility for factors within their control.
Accepting negative feedback too quickly can also send the wrong message to others.

4.4.3 Superficial vs. Genuine Engagement

People who are superficially engaged seem open to negative feedback, willing to admit
failure, and solicit advice for future improvement. In reality, however, they are purposefully
attempting to gain acceptance and avoid further confrontation about the problem.
Superficially engaged individuals may benefit from broadening their base of feedback and
develop a list of action items they are willing to take on to improve. More importantly, they
should make sure to have an accountability structure set up for individual improvement.

Conversely, genuinely engaged individuals tend to actively participate in feedback sessions


about bad decisions and commit themselves to finding ways to improve future decisions.
Although this response is preferable to superficial engagement, genuinely engaged
individuals may also benefit from thinking about times when their desire to improve may
have resulted in their acting on feedback that may not have been constructive or even valid.
They may consider what they can do to evaluate the quality of post-decision feedback they
receive before acting on it.

4.5 Feedback Response Patterns

As previously noted, people vary in their reactions to feedback by (a) becoming defensive or
remaining cool-headed, (b) denying or accepting criticism, and (c) superficially or genuinely
engaging with the feedback. However, these reactions do not exist in isolation. Instead,
each person exhibits a combination of reactions that define a response pattern. For
example, one person may react defensively, deny problems, and superficially engage in
feedback, whereas a different individual may remain cool-headed, accept criticism, and
genuinely engage in feedback. The possible combinations of response tendencies based on
these scales allows us to define eight feedback response styles. These styles provide a
more holistic account of each persons typical reactions to negative feedback. However,
unlike our eight decision-making styles, these eight feedback response patterns are not
described by archetype occupations; rather, we describe these eight feedback response

60

4. INTERPRETATION & USES

patterns in the simplest terms, using the scores comprising each pattern. We describe each
pattern below.

4.5.1 Defensive, Denial, Superficial

Some individuals respond to negative feedback by becoming defensive, denying or


deflecting criticism, and superficially engaging in feedback sessions. Such individuals tend
to blame undesirable outcomes on other people or external factors, react emotionally to bad
news, and avoid taking responsibility for unwelcome results. They may become annoyed
with negative feedback and/or quickly forget it, inadvertently train others not to deliver bad
news, have difficulty learning from experience, and resist coaching and other development
efforts.

4.5.2 Defensive, Acceptance, Superficial

Some people tend to respond to negative feedback by becoming defensive at first, then
accepting the criticism, but only superficially engaging in feedback. They are initially upset
by negative feedback, seeing it as a form of rejection. They defend themselves by blaming
other people and external circumstances for the mistakes. However, after the initial
resistance has passed, they may see the negative feedback as legitimate. Nonetheless, due
to their superficial engagement, they often avoid taking complete responsibility for
undesirable outcomes, pretend to agree with feedback to get along, and have difficulties
changing their behavior following feedback.

4.5.3 Defensive, Denial, Genuine

Some individuals respond to negative feedback by becoming defensive, denying or


deflecting criticism, but ultimately engaging in feedback. Such people tend to initially react
emotionally, blame other people, deflect or deny any mistakes, and have difficulties
accepting criticism. Ironically, in the context of denying any problems, they will actively
participate in feedback sessions and may benefit from coaching after the initial emotional
reaction has subsided.

4.5.4 Defensive, Acceptance, Genuine

Other people tend to respond to negative feedback by becoming defensive at first, then
accepting the criticism and genuinely engaging in feedback sessions. Such individuals tend
to first react defensively and emotionally to negative feedback and blame other people or
external factors for the mistakes. However, once the initial emotional response has passed,
they may be more modest and straightforward in considering feedback, willing to reflect on
past mistakes, remain genuinely engaged in feedback sessions, appear motivated to
improve performance and change behavior, and benefit from coaching.

4.5.5 Cool-Headed, Denial, Superficial

Some individuals respond calmly to negative feedback, but deny criticism and superficially
engage in feedback. Such individuals tend to receive bad news calmly, are confident in their

61

4. INTERPRETATION & USES

decision-making, lack a sense of urgency to correct past mistakes, disregard negative


feedback, prefer not to dwell on past mistakes, show skill in reinterpreting failure as success,
have difficulties learning from experience, and avoid taking personal responsibility for failed
decisions.

4.5.6 Cool-Headed, Denial, Genuine

Other people tend to respond to negative feedback by remaining calm and genuinely
engaging in feedback, but denying criticism. Such individuals tend to remain composed in
the face of bad news, rarely take bad news personally, and deflect or deny feedback with
which they disagree. However, they will genuinely engage in feedback sessions and may
benefit from coaching once a trusting environment is formed that encourages them to safely
acknowledge their mistakes.

4.5.7 Cool-Headed, Acceptance, Superficial

Some individuals respond to negative feedback by remaining calm and accepting criticism,
but only superficially engaging in feedback. Such people tend to take bad news well, avoid
taking negative feedback personally, accept past failures, avoid dwelling on the past, and
appear willing to engage in feedback sessions. They will seem to listen to and agree with
feedback, but may have difficulties actually changing future behavior based on this coaching.

4.5.8 Cool-Headed, Acceptance, Genuine

Finally, other people tend to respond to negative feedback by remaining cool-headed,


accepting criticism, and genuinely engaging in feedback sessions. Such individuals tend to
remain composed in the face of bad news, accept negative feedback in an open-minded and
non-defensive manner, accept personal responsibility for past mistakes, directly address
previous failures to improve future decisions, genuinely engage in feedback sessions, and
carefully reflect on feedback. As this description suggests, these individuals can most easily
benefit from coaching because their natural response tendencies do not create internal
obstacles.

4.6 Openness to Feedback and Coaching

As noted above, people demonstrate a variety of reactions to negative feedback, many of


which are counterproductive. With combinations of the pre-decision approach scales, recall
that our interest was in using those eight combinations to describe distinct decision-making
styles. However, with the response to feedback scales, we are less interested in defining
eight response styles and more interested in describing how each individuals combination
of scores across these scales define his or her overall openness to feedback and coaching.

By combining peoples scores across the three feedback scales, we can describe their
overall openness to feedback and coaching to improve future decisions. For example, one
person may generally resist coaching, whereas another individual may be generally receptive
to these efforts. This is crucial information because it concerns the likelihood that a person
will be able to learn from his or her mistakes to improve future business decisions. Put

62

4. INTERPRETATION & USES

another way, this information concerns a persons coachability, a crucial element of both
good judgment and effective performance. In general, individuals can be characterized as
resistant, neutral, or receptive to feedback and coaching. Although being receptive is
generally preferable, each group has its own set of strengths and challenges. We describe
each below.

4.6.1 Resistant

Feedback resistant individuals may be well-suited for making difficult decisions and
standing by them despite criticism. However, unmitigated resistance to feedback and
coaching over time may prevent others from sharing candid feedback, erode team dynamics,
and impede effective performance.

Individuals characterized as feedback resistant are not un-coachable, but must


consistently monitor their counterproductive reactions to ensure that they mitigate them to
change their behavior and improve future decisions. As such, in response to feedback,
resistant individuals should focus on not taking criticism personally or reflexively denying or
deflecting feedback. Instead, they should take time to carefully consider the feedback
before responding. If they can think more deeply about their role in these mistakes, they
may be able to find solutions that will help them make better decisions in the future. To
facilitate these conditions, it is important to establish a safe and trusting feedback
environment. For example, coaches may need to spend more time up front establishing
credibility when working with resistant individuals. It is also important to set realistic
expectations and provide positive reinforcement when these individuals act on feedback.

4.6.2 Neutral

Feedback neutral individuals are often good at recognizing instances where they may be
responsible for bad decisions. However, they may alternate between calm and emotional
reactions, acceptance and denial of criticism, and active and inactive engagement in
feedback sessions. In short, these individuals demonstrate a middle of the road response
to feedback, neither resisting it nor accepting responsibility entirely.

In response to negative feedback, neutral individuals should carefully consider times when
they may have avoided or deflected feedback before giving it the full consideration it
deserved. They should also take time to explore the feedback and their role in past
mistakes before responding. In addition, they may benefit from reflecting on times when
they could have more actively participated in feedback sessions. Insights gained from these
efforts may potentially help such individuals take full advantage of future opportunities to
improve decision-making.

4.6.3 Receptive

Feedback receptive individuals seem willing to accept fault for their failed decisions and
benefit from coaching. However, their tendency to internalize fault may also cause them to
take responsibility for others errors as well as external factors beyond their control. As such,

63

4. INTERPRETATION & USES

these individuals should carefully monitor their reactions to feedback to ensure that they are
not internalizing excess responsibility for undesirable outcomes.

Feedback receptive individuals should also consider what roles they and others have played
in past failed decisions. These reflections should help them focus on feedback about their
performance without taking blame for others mistakes. Also, instead of automatically
internalizing all feedback, they should consider feedback on a case-by-case basis to
determine its merit. Through these efforts, feedback receptive individuals can leverage
constructive disagreement and debate to further explore solutions to previous mistakes and
improve future business decisions.

64

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

5. Administering the Hogan Judgment Report

Hogan Assessment Systems, Inc. (Hogan) provides a state-of-the-art administration


platform developed to meet the needs of our clients. Hogan designed this platform to
maximize security, ease of use, speed, and flexibility. This allows us to accommodate clients
of all sizes. An overview of key features of this system is presented below. For further
information, please contact Hogans Customer Service Department at 1-800-756-0632 or
customerservice@hoganassessments.com. Office hours are 8am-5pm U.S. central time and
after-hours messages are checked daily.

5.1 Key Features of the Online Assessment

It is important that test administrators: (a) understand how participants complete an


assessment, (b) answer questions or concerns participants may raise, and (c) use test
administrator tools effectively. To address these issues, Hogan trains administrators in the
functionality of the online assessment system. In the initial training session, we instruct new
administrators on how to create participant IDs and use other tools on the administrative
website. Additional training is available for creating participant groups, obtaining reports,
changing report options, and specifying report delivery options.

The Hogan testing system is fully redundant, using multi-location systems architecture to
ensure constant availability. Clients can access the testing platform 24 hours a day, 7 days
a week from any internet-enabled device. Test results are normally delivered in 90 seconds
or less, making results nearly instantaneous. The system delivers results via the web or
encrypted PDF files sent by e-mail. Hogan consults with outside security experts to ensure
data security. We use 2,048-bit secure access via password protection when safeguarding
clients and users assessment data. All Hogan web ordering systems allow us to tailor the
ordering and reporting experience to each user based on a hierarchical system of client and
user preferences.

Whether a client orders from a single office or locations across the globe, all orders can flow
through a single account. Hogans product-level security features allow clients to restrict
individual users ability to order and view reports on a product-by-product basis.

5.2 Completing the Assessment Using the Online Platform

Although test taking time may vary depending on a users reading speed, the Hogan
Judgment Assessment usually requires about 20 minutes to complete. Once a participant
receives a Hogan ID from the administrator, he/she logs onto the website at
http://www.gotohogan.com or a customized portal designed specifically for a Hogan client.
To log onto the website, a minimum version of Microsoft Internet Explorer 7, Chrome, Safari,
Firefox or other browser that supports cookies and JavaScript is needed. Once on the
website, the individual sees a login page similar to the one presented in Figure 5.1.

65

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.1
Participant Login Web Page

On the login page, the participant is asked to enter his/her assigned Hogan ID and password
(e.g., BB123456 and SAMPLE) and click the Go button. The participant is then prompted
to complete a brief set of demographics questions (see Figure 5.2) and agree to an informed
consent clause (see section 5.3). This clause outlines information regarding the purpose,
administration, and results of the assessment.

66

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.2
Participant Information Web Page

Once users have logged in, the system asks them to create a personal password and
complete additional information fields. When all fields are complete, they must click Go
and are directed to the participant menu page. This page displays each assessment they
must complete (see Figure 5.3). After users complete an assessment, the system returns
them to this page if they need to select and complete additional assessments.

67

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.3
Participant Menu Wed Page

5.2.1 Numerical Section

The numerical section of the Hogan Judgment Assessment consists of 15 multiple-choice


numerical problems. Test takers have 10 minutes to complete as many of these problems
as they can. Participants can use scratch paper and a pencil or pen when completing this
section, but cannot use calculators or other external devices. Figure 5.4 presents an
example of instructions from the numerical section.

68

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.4
Numerical Section Instructions

Following the instructions, participants receive two sample questions to familiarize them
with the format of the items. For each sample item, the system also indicates the correct
response. Figure 5.5 presents an example of a numerical sample item.

Once participants complete the sample questions, they can select Next to start the
numerical section or Stop Assessment to quit. After they begin this section, they cannot
go back and finish it at a later time.

69

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.5
Numerical Section Sample Item

Once participants start the numerical section, they can click Next or Back to navigate
between questions. They can also skip questions and go back to review previous questions
before their time runs out. An on-screen timer displays the time remaining and flashes red
to warn participants when they are nearly out of time. Figure 5.6 presents an example of a
page from the numerical section.

70

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.6
Numerical Section Sample Page

If participants complete all questions with time remaining, they will see a question review
page. From this page, they can revisit skipped questions or review questions they previously
answered. The review page presents buttons for each of the 15 items, with unanswered
questions appearing in red. If a participant selects a number, the system redirects them to
that question. Answering that question or selecting Next returns the participant to the
review page. Figure 5.7 presents an example of the review page.

71

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.7
Numerical Section Review Page

If the 10-minute timer runs out before participants answer all questions, the system
automatically saves their answers and provides a message instructing them that their time
has expired and their responses have been recorded (see Figure 5.8). Participants click
Next on this message to proceed to the next section of the Hogan Judgment Assessment.

72

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.8
Out of Time Page

5.2.2 Verbal Section

The verbal section of the Hogan Judgment Assessment contains 48 statements pertaining to
letters in the alphabet, analogies, types of items, opposites, sizes, and meanings, and asks
participants to indicate if the statements are true or false. Participants have two minutes to
respond to as many statements as they can. Figure 5.9 presents an example of instructions
from the verbal section.

73

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.9
Verbal Section Instructions

Following the instructions, participants view two sample questions to familiarize them with
the format of the items. For each sample item, the system also indicates the correct
response. Figure 5.10 presents an example of a verbal sample item. Once participants
complete the sample questions, they can select Next to start the verbal section or Stop
Assessment to quit. After they begin this section, they cannot go back and finish it at a
later time.

74

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.10
Verbal Section Sample Item

Once participants start the verbal section, they can click Next or Back to navigate
between questions. They can also skip questions and go back to review previous questions
before their time runs out. An on-screen timer displays the time remaining and flashes red
to warn participants when they are nearly out of time. Figure 5.11 presents an example of a
page from the verbal section.

75

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.11
Verbal Section Sample Page

If participants complete all questions with time remaining, they will see a question review
page. From this page, they can revisit skipped questions or review questions they previously
answered. The review page presents buttons for each of the 48 items, with unanswered
questions appearing in red. If a participant selects a number, the system redirects them to
that question. Answering that question or selecting Next returns the participant to the
review page. Figure 5.12 presents an example of the review page.

76

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.12
Verbal Section Review Page

If the two-minute timer runs out before participants answer all questions, the system
automatically saves their answers and provides a message instructing them that their time
has expired and their responses have been recorded (see Figure 5.8). Participants click
Submit on this message to proceed to the next section of the Hogan Judgment
Assessment.

77

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

5.2.3 Decision-Making Style Section

The decision-making style section of the Hogan Judgment Assessment contains 75


true/false statements. This section is untimed and has no correct or incorrect responses.
Instead, participants read each statement and respond True if that statement accurately
describes them or False if it does not. Although participants are not required to answer
every question, scores cannot be computed and results will not be presented if the
participant skips more than one third of the items on any scale. Figure 5.13 presents an
example of instructions from the decision-making style section.

Figure 5.13
Decision-Making Style Section Instructions

Following the instructions, participants receive two sample questions to familiarize them
with the format of the items. For each sample item, the system also provides feedback to
reiterate the fact that this section has no correct or incorrect responses and responses
reflect general preferences. Figure 5.14 presents an example of a sample item from the
decision-making style section.

78

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.14
Decision-Making Style Section Sample Item

Once participants start the decision-making style section, they can click Next or Back to
navigate between questions. They can also skip questions and go back to review previous
questions. Figure 5.15 presents an example of a page from the decision-making style
section.

79

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

80

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.15
Decision-Making Style Section Sample Page

Once participants complete all questions, they will see a question review page. From this
page, they can revisit skipped questions or review questions they previously answered. The
review page presents buttons for each of the 75 items, with unanswered questions
appearing in red. If a participant selects a number, the system redirects them to that
question. Answering that question or selecting Next returns the participant to the review
page. Figure 5.16 presents an example of the review page.

81

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Figure 5.16
Decision-Making Style Section Review Page

Participants can start and stop this section of the assessment at their convenience. If they
stop the assessment, the system retains all information submitted to that point. They can
log back into the system with their user ID and self-created personal password to continue
the assessment at any time. Once they are satisfied with their responses, they can submit
them to complete the last section of the Hogan Judgment Assessment.

A scoring engine automatically processes the participants responses and sends reports to
one or more e-mail addresses as designated by the administrator. If the account
administrator or participant experiences a problem, they can contact Hogans Customer
Service Department at 1-800-756-0632 or customerservice@hoganassessments.com.

5.3 Participant Informed Consent

Hogan requires that all individuals taking its assessments give their informed consent to
participate in the assessment process. This is the fundamental concept that underlies all
current and anticipated data protection protocols and legislation. In order for individuals
taking the assessments to give their informed consent, they must understand the purpose
of the assessment, the likely use of the assessment data, and how the data are protected.
These protocols are described below and are binding on all Hogan clients and individuals
taking the assessments. Failure to comply with any of these safeguards will constitute
grounds for termination of any agreements in place between Hogan and the person(s) or
entity(ies) concerned.

82

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

5.3.1 Consent

By initiating the assessment(s), the participant expressly consents to the processing,


including storage, copying, deletion, printing, transfer, aggregation and use, of the data
he/she provides as set forth below. This consent is given voluntarily and the participant may
withdraw such consent at any time. Hogan shall not be responsible for or subject to any
liability resulting from any change to the participants potential or continued employment
status that may result from either (a) their taking of the assessment(s), or (b) their decision
to withdraw their consent to the processing of the data he/she provides.

5.3.2 Purpose

Hogans assessments were created to provide personal characteristic information and


feedback to trained and accredited consultants and HR professionals. The data participants
provide, including the replies they submit during the assessment(s), will be processed by
Hogan. The results of the assessment(s) are primarily used for selection and/or
development by the requesting organization.

5.3.3 Data Processing

Hogan has the right to process the data participants provide, including the storage, copying,
deletion, printing, transfer, aggregation and use thereof. Hogan may also transfer, via
secured methods, the data to trained and accredited consultants or HR professionals for
selection and/or development purposes. In addition, Hogan may use anonymously held (i.e.,
identifying information removed) aggregated data for its own research purposes. All Hogan
clients are responsible for complying with national and international protocols covering data
use and storage. For additional information on Hogans data privacy policy and its
compliance with the U.S.-EU Safe Harbor and the U.S.-Swiss Safe Harbor Frameworks,
please visit www.hoganassessments.com/privacy-policy for more information.

5.3.4 Access to Data

The dissemination of assessment results is the sole responsibility of the requesting


organization. Hogan will only provide results to individuals taking the assessments under
specific direction from the requesting organization. Individuals taking the assessments are
not guaranteed access to their individual results. However, they may withdraw their consent
to the processing of the data they provided at any time, and such data will be deleted from
Hogans systems. If participants choose to withdraw their consent to the processing of the
data they provide after Hogan has provided their assessment results to the requesting
organization, Hogan shall have no obligation to ensure the deletion of any data that may be
contained in such assessment results. In addition, Hogan shall not be responsible for or
subject to any liability resulting from any change to the participants potential or continued
employment status that may result from their decision to withdraw their consent.

83

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

5.3.5 Security

In order to safeguard assessment results, the website contains only the assessment items,
not the assessment programs (which are held by Hogan and its clients). It is impossible to
process assessment results through the website. Assessment results can only be processed
by downloading the raw data, decrypting it, and scoring these data with appropriate
programs. Until that time, responses to assessment items are merely encrypted
alphanumeric strings with no discernible meaning.

5.3.6 Contact

If participants have any questions about informed consent, the purpose of the
assessment(s), the processing of the data they provide, or Hogans privacy policy, they can
contact customerservice@hoganassessments.com or their local Hogan distributor.

5.4 Accommodating Individuals with Disabilities

The Americans with Disabilities Act of 1990 (ADA) prohibits employment discrimination
against qualified individuals with disabilities. This law has important implications for
employers procedures used in interviewing, testing, and hiring new employees. For pre-
employment testing, the ADA specifies that employers must provide alternate forms of
employment testing that accurately (assess) the skills, aptitudes, or whatever other factor
of such applicant or employee that such test purports to measure, rather than reflecting the
impaired sensory, manual or speaking skills of such employee or applicant Sec. 102(b)(7),
42 U.S.C.A.Sec. 12112. Hogan complies with the ADA requirements by working with clients
to accommodate individuals with special needs. Large print assessments are available from
Hogans Customer Service Department at 1-800-756-0632 or by e-mail at
customerservice@hoganassessments.com. Hogan can make additional accommodations
on a case-by-case basis.

5.5 Frequently Asked Questions

The following section presents frequently asked questions and answers to those questions.

Q. I am trying to sign back in to complete the assessment but my user ID and password are
not working.
A. On the initial participant information screen, you were asked to change the initial
password you were given to a personal password you could more easily remember. Please
use the new personal password you created to log back into the system.

Q. Is it a timed assessment?
A. The numerical and verbal sections of the Hogan Judgment Assessment have ten- and two-
minute timers, respectively. The decision-making style section is untimed and you can take
as much time as needed to complete this section.

84

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

Q. Can I stop the assessment at any time?


A. Once you begin the numerical and verbal sections of the assessment, you must complete
them. However, during the decision-making style section you may stop at any time and
return at your convenience. You will need your user ID and new personal password to log
back into the website.

Q. How long will the assessment take?


A. We anticipate that it should take you about 20 minutes to complete the Hogan Judgment
Assessment.

Q. Will I receive a copy of my results?


A. We are not at liberty to share or discuss results with participants. Results are sent to the
company that asked you to complete the assessment. That company decides whether or
not to share results with you.

Q. Will all my data be lost if my system freezes before I complete the assessment?
A. No, your responses are saved after each page is completed.

85

REFERENCES

REFERENCES

Ackerman, P. L., & Heggestad, E. D. (1997). Intelligence, personality, and interests: Evidence
for overlapping traits. Psychological Bulletin, 121, 219.

Aguinis, H., Henle, C. A., & Ostroff, C. (2001). Measurement in work and organizational
psychology. Handbook of industrial, work and organizational psychology, 1, 27-50.

Aitkenhead, M. (1984). Impression-management and consistency effects in the processing


of feedback. British Journal of Social Psychology, 23, 213-222. doi:10.1111/j.2044-
8309.1984.tb00632.x

American Educational Research Association, American Psychological Association, & the


National Council on Measurement in Education (1999). Standards for educational
and psychological testing. Washington, D.C.: American Educational Research
Association.

American Psychiatric Association (APA). (2013). Diagnostic and statistical manual of mental
disorders (5th ed.). Washington, DC: Author.

Americans with Disabilities Act of 1990, 102(b)(7), 42 U.S.C.A. 12112.

Ariely, D. (2008). Predictably irrational: The hidden forces that shape our decisions. New
York, NY: HarperCollins.

Aronson, E., Wilson, T. D., & Akert, R. M. (2010). Social Psychology. Upper Saddle River, NJ:
Prentice Hall.

Ashford, S. J., & Northcraft, G. B. (1992). Conveying more (or less) than we realize: The role
of impression-management in feedback-seeking. Organizational Behavior and
Human Decision Processes, 53, 310-334.

Atkinson, J. W. (1957). Motivational determinants of risk-taking behavior. Psychological


Review, 64, 359-372. doi:10.1037/h0043445

Axelrod, A. (2006). Profiles in audacity: Great decisions and how they were made. New
York, NY: Sterling Publishing Company, Inc.

Baddeley, A. D. (1986). Working memory. Oxford: Clarendon.

Baddeley, A. D. (1997). Human memory: Theory and practice. Psychology Press.

Barrick, M. R., Stewart, G. L., & Piotrowski, M. (2002). Personality and job performance: Test
of the mediating effects of motivation among sales representatives. Journal of
Applied Psychology, 87, 4351.

86

REFERENCES

Baumeister, R., Bratslavsky, E., Finkenauer, C., & Vohs, K. D. (2001). Bad is stronger than
good. Review of General Psychology, 5, 323370.

Bechara, A. (2005). Decision-making, impulse control and loss of willpower to resist drugs: a
neurocognitive perspective. Nature Neuroscience, 8, 1458-1463.

Bell, S. T., & Arthur, W. R. (2008). Feedback acceptance in developmental assessment


centers: The role of feedback message, participant personality, and affective
response to the feedback session. Journal of Organizational Behavior, 29, 681-703.
doi:10.1002/job.525

Benartzi, S., & Thaler, R. H. (1995). Myopic loss aversion and the equity premium puzzle.
Quarterly Journal of Economics, 110, 73.

Benet-Martinez, V., & John, O. (1998). Los Cinco Grandes across cultures and ethnic groups:
Multitrait-multimethod analyses of the Big Five in Spanish and English. Journal of
Personality and Social Psychology, 75, 729-750.

Berry, C. M., Ones, D. S., & Sackett, P. R. (2007). Interpersonal deviance, organizational
deviance, and their common correlates: a review and meta-analysis. Journal of
Applied Psychology, 92, 410.

Betsch, C. (2008). Chronic preferences for intuition and deliberation in decision-making:


Lessons learned about intuition from an individual differences approach. In H.
Plessner, C. Betsch, T. Betsch (Eds.), Intuition in Judgment and Decision-making (pp.
231-248). Mahwah, NJ US: Lawrence Erlbaum Associates Publishers.

Borman, W. C., Penner, L. A., Allen, T. D., & Motowidlo, S. J. (2001). Personality predictors of
citizenship performance. International Journal of Selection and Assessment, 9, 52-
69.

Brett, J. F., Atwater, L. A. (2001). 360-degree feedback: Accuracy, reactions and perceptions
of usefulness. Journal of Applied Psychology, 86, 930942.

British Psychological Society (2007). Hogan Personality Inventory. British Psychological


Society Psychological Testing Centre test reviews. London, UK: Author.

Brocas, I. (2012). Information processing and decision-making: Evidence from the brain
sciences and implications for economics. Journal of Economic Behavior and
Organization, 83, 292-310. doi:10.1016/j.jebo.2012.06.004

Bruine de Bruin, W., Parker, A. M., & Fischhoff, B. (2007). Individual differences in adult
decision-making competence. Journal of Personality and Social Psychology, 92, 938.

Bruine de Bruin, W., Del Missier, F., & Levin, I. P. (2012). Individual differences in decision-
making competence. Journal of Behavioral Decision-making, 25, 329-330.

87

REFERENCES

Busenitz, L. W., & Barney, J. B. (1997). Differences between entrepreneurs and managers in
large organizations: Biases and heuristics in strategic decision-making. Journal of
Business Venturing, 12, 9-30.

Button, S., Mathieu, J., & Zajac, D. (1996). Goal orientation in organizational research: A
conceptual and empirical foundation. Organizational Behavior and Human Decision
Processes, 67, 26-48.

Campbell, D. P., Hyne, S. A., & Nilsen, D. L. (1992). Campbell Interest and Skill Survey
manual. Minneapolis, MN: National Computer Systems, Inc.

Carver, C. S., & White, T. L. (1994). Behavioral inhibition, behavioral activation, and affective
responses to impending reward and punishment: the BIS/BAS scales. Journal of
Personality and Social Psychology, 67, 319.

Chamorro-Premuzic, T. (2008). The numerical reasoning test 20-items (NRT-20). London:


Goldsmiths, University of London (unpublished test).

Chen, G., Crossland, C., & Luo, S. (2014). Making the same mistake all over again: CEO
overconfidence and corporate resistance to corrective feedback. Strategic
Management Journal. Forthcoming.

Choudhry, N. K., Fletcher, R. H., & Soumerai, S. B. (2005). Systematic review: The
relationship between clinical experience and quality of health care. Annals of Internal
Medicine, 142, 260273.

Citroen, C. L. (2011). The role of information in strategic decision-making. International


Journal of Information Management, 31, 493-501.
doi:10.1016/j.ijinfomgt.2011.02.005

Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in scale development.
Psychological Assessment, 7, 309-319.

Cloninger, C., Przybeck, T., Svrakic, D., & Wetzel, R. (1994). TCIthe temperament and
character inventory: A guide to its development and use. St Louis (MO): Center for
Psychobiology of Personality, Washington University.

Conn, S. R., & Rieke, M. L. (1994). The 16PF fifth edition technical manual. Champaign, IL:
Institute for Personality and Ability Testing.

Cooper, A. C., Woo, C. Y., & Dunkelberg, W. C. (1988). Entrepreneurs' perceived chances for
success. Journal of Business Venturing, 3, 97-108.

Costa, P. T. Jr., & McCrae, R. R. (1992). Revised NEO Personality Inventory (NEO-PI-R) and
NEO Five-Factor Inventory (NEO-FFI): Professional manual. Odessa, FL: Psychological
Assessment Resources.

88

REFERENCES

Davis, C., Patte, K., Tweed, S., & Curtis, C. (2007). Personality traits associated with
decision-making deficits. Personality and Individual Differences, 42, 279290.

De Neys, W., Novitskiy, N., Geeraerts, L., Ramautar, J., & Wagemans, J. (2011). Cognitive
control and individual differences in economic ultimatum decision-making. PloS
one, 6, e27107. doi:10.1371/journal.pone.0027107

De Raad, B., & Perugini, M. (Eds.). (2002). Big Five assessment. Seattle, WA: Hogrefe &
Huber.

DeVillis, R. F. (1991). Scale development: Theory and application. Newbury Park, CA: Sage.

DeVries, D. L. (1992). Executive selection: Advances but no progress. Issues and


Observations, 12, 1-5.

Dewberry, C., Juanchich, M., & Narendran, S. (2013). Decision-making competence in


everyday life: The roles of general cognitive styles, decision-making styles and
personality. Personality and Individual Differences, doi:10.1016/j.paid.2013.06.012

Donovan, J. J., & Williams, K. J. (2003). Missing the mark: Effects of time and causal
attributions on goal revision in response to goal-performance discrepancies. Journal
of Applied Psychology, 88, 379390.

Dudek, F. J. (1979). The continuing misinterpretation of the standard error of measurement.


Psychological Bulletin, 86, 335-337.

Dudley, N. M., Orvis, K. A., Lebiecki, J. E., & Cortina, J. M. (2006). A meta-analytic
investigation of conscientiousness in the prediction of job performance: Examining
the intercorrelations and the incremental validity of narrow traits. Journal of Applied
Psychology, 91, 4057.

Elliot, A. J., & Thrash, T. M. (2002). Approach-avoidance motivation in personality: approach


and avoidance temperaments and goals. Journal of Personality and Social
Psychology, 82, 804.

Equal Employment Opportunity Commission, Civil Service Commission, Department of Labor,


& Department of Justice (1978). Uniform guidelines on employee selection
procedures. Federal Register, 43, 38290-38315.

Feather, N. T. (1968). Change in confidence following success or failure as a predictor of


subsequent performance. Journal of Personality and Social Psychology, 9, 38.

Finucane, M. L., Alhakami, A., Slovic, P., & Johnson, S. M. (2000). The affect heuristic in
judgments of risks and benefits. Journal of Behavioral Decision-making, 13, 1-17.

89

REFERENCES

Fischhoff, B. (1982). Debiasing. In D. Kahneman, P. Slavic, & A. Tversky (Eds.), Judgment


under uncertainty: Heuristics and biases. New York: Cambridge Univ. Press.

Freud, A. (1946). The ego and the mechanisms of defense. New York: International
Universities Press. (Original work published 1936).

Gallo, A. (2012, May 15). How to get feedback when you're the boss [Web log message].
Retrieved from https://hbr.org/2012/05/how-to-get-feedback-when-youre

Gigerenzer, G., & Gaissmaier, W. (2011). Heuristic decision-making. Annual Review of


Psychology, 62, 451-482.

Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring


the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. De
Fruyt, & F. Ostendorf (Eds.), Personality psychology in Europe (Vol. 7, pp. 7-28).
Tilburg, Netherlands: Tilburg University Press.

Goldberg, L. R. (2008). The Eugene-Springfield Community Sample: Information available


from the research participants. Oregon Research Institute Technical Report Vol. 48,
No. 1. Eugene, Oregon: Oregon Research Institute.

Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C.R., & Gough,
H.C. (2006). The International Item Pool and the future of public-domain personality
measures. Journal of Research in Personality, 40, 84-96.

Gough, H. G. (1975). Manual for the California Psychological Inventory. Palo Alto, CA:
Consulting Psychologists Press.

Gough, H. G. (1996). CPI manual. Palo Alto, CA: Consulting Psychologists Press.

Gruys, M. L., & Sackett, P. R. (2003). Investigating the dimensionality of counterproductive


work behavior. International Journal of Selection and Assessment, 11, 30-42.

Hamers, J. H., Ruijssenaars, A. J. J. M., & Sijtsma, K. (Eds.). (1992). Learning potential
assessment. CRC Press.

Harnish, V. (2012). The greatest business decisions of all time: How Apple, Ford, IBM,
Zappos, and others made radical choices that changed the course of business. New
York, NY: Fortune Books.

Hodgkinson, G. P., Maule, A. J., Bown, N. J., Pearman, A. D., & Glaister, K. W. (2002). Further
reflections on the elimination of framing bias in strategic decision-making. Strategic
Management Journal, 23, 1069-1076.

Hogan Assessment Systems (2014). Hogan Development Survey form 5 technical


supplement. Tulsa, OK: Hogan Assessment Systems.

90

REFERENCES

Hogan, J., Barrett, P., & Hogan R. (2007). Personality measurement, faking, and employment
selection. Journal of Applied Psychology, 92, 1270-1285.

Hogan, J., & Hogan, R. (2010). Motives, Values, Preferences Inventory manual: 2010
administrative and norming updates. Tulsa, OK: Hogan Press.

Hogan, J., & Holland, B. (2003). Using theory to evaluate personality and job-performance
relations: A socioanalytic perspective. Journal of Applied Psychology, 88, 100-112.

Hogan, R. (1983). A socioanalytic theory of personality. In M. M. Page (Ed.), 1982 Nebraska


symposium on motivation (pp. 55-89). Lincoln: University of Nebraska Press.

Hogan, R. (1991). Personality and personality measurement. In M. D. Dunnette & L. M.


Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 2, 2nd ed.,
pp. 327-396). Palo Alto, CA: Consulting Psychologists Press.

Hogan, R. (1996). A socioanalytic perspective on the five-factor model. In J. S. Wiggins (Ed.),


The five-factor model of personality (pp.163-179). New York: Guilford.

Hogan, R., & Hogan, J. (1995). Hogan Personality Inventory manual (2nd ed.). Tulsa, OK:
Hogan Assessment Systems.

Hogan, R., & Hogan, J. (2007). Hogan Personality Inventory manual (3rd ed.). Tulsa, OK:
Hogan Assessment Systems.

Hogan, R., & Hogan, J. (2009). Hogan Development Survey manual (2nd ed.). Tulsa, OK:
Hogan Press.

Hogan, R., Hogan, J., & Roberts, B. W. (1996). Personality measurement and employment
decisions: Questions and answers. American Psychologist, 51, 469-477.

How employees process feedback [Web log message]. (2013, March 28). Retrieved from
http://blog.talkdesk.com/how-employees-process-feedback

Hurtz, G. M., & Donovan, J. J. (2000). Personality and job performance: the Big Five
revisited. Journal of Applied Psychology, 85, 869.

Ilgen, D. R., Fisher, C. D., & Taylor, M. S. (1979). Consequences of individual feedback on
behavior in organizations. Journal of Applied Psychology, 64, 349.

Ilies, R., & Judge, T. A. (2005). Goal regulation across time: The effects of feedback and
affect. Journal of Applied Psychology, 90, 453467.

Ilies, R., Judge, T. A., & Wagner, D. T. (2010). The influence of cognitive and affective
reactions to feedback on subsequent goals: Role of behavioral
inhibition/activation. European Psychologist, 15, 121.

91

REFERENCES

Jackson, D. N. (1984). Multidimensional aptitude battery. Port Huron, MI: Research


Psychologists Press.

Jackson, D. N. (1994). Jackson Personality Inventory-revised. Sigma Assessment Systems,


Research Psychologists Press Division.

Jackson, D. N., Paunonen, S. V., Fraboni, M., & Goffin, R. D. (1996). A five-factor versus six-
factor model of personality structure. Personality and Individual Differences, 20, 33-
45.

Jackson, C. J. (2003). Gray's reinforcement sensitivity theory: A psychometric critique.


Personality and Individual Differences, 34, 533-544.

John, O., & Srivastava, S. (1999). The Big Five Trait taxonomy: History, measurement, and
theoretical perspectives. In Pervin, L. A., & John, O. P. (Eds.), Handbook of Personality:
Theory and Research (2nd ed.) (pp. 102-138). New York: Guilford Press.

Kahneman, D. (2003). Maps of bounded rationality: Psychology for behavioral


economics. The American Economic Review, 93, 1449-1475.

Kahneman, D., Lovallo, D., & Sibony, O. (2011). Before you make that big decision. Harvard
Business Review, 89, 50-60.

Kauer, D., Waldeck, T., & Schffer, U. (2007). Effects of top management team
characteristics on strategic decision-making: Shifting attention to team member
personalities and mediating processes. Management Decision, 45, 942-967.
doi:10.1108/00251740710762017

Kliger, D., & Kudryavtsev, A. (2010). The availability heuristic and investors' reaction to
company-specific events. The Journal of Behavioral Finance, 50-65.

Kliger, D., & Levit, B. (2009). Evaluation periods and asset prices: Myopic loss aversion at
the financial marketplace. Journal of Economic Behavior and Organization, 71, 361
371.

Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A
historical review, a meta-analysis, and a preliminary feedback intervention theory.
Psychological Bulletin, 119, 254-284.

Kluger, A. N., & DeNisi, A. (1998). Feedback interventions: Toward the understanding of a
double-edged sword. Current Directions in Psychological Science, 7, 67-72.

Kouzes, J., & Posner, B. (2014, February 27). To get honest feedback, leaders need to ask
[Web log message]. Retrieved from https://hbr.org/2014/02/to-get-honest-
feedback-leaders-need-to-ask/

92

REFERENCES

Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficultires in recognizing
ones own incopmetence lead to inflated self-assessments. Journal of Personality
and Social Psychology, 77, 1121-1134.

Kruger, J., Wirtz, D., Van Boven, L., & Altermatt, T. W. (2003). The effort heuristic. Journal of
Experimental Social Psychology, 91-98.

Latham, G. P., & Locke, E. A. (2007). New developments in and directions for goal-setting
research. European Psychologist, 12, 290-300.

Lee, K., & Ashton, M. C. (2004). Psychometric properties of the HEXACO Personality
Inventory. Multivariate Behavioral Research, 39, 329-358.

Leonard, D., & McAdam, R. (2002). The role of the business excellence model in operational
and strategic decision-making. Management Decision, 40, 17-25.

LePine, J. A., Erez, A., & Johnson, D. E. (2002). The nature and dimensionality of
organizational citizenship behavior: a critical review and meta-analysis. Journal of
Applied Psychology, 87, 52.

Levin, I. P., Gaeth, G. J., Schreiber, J., & Lauriola, M. (2002). A new look at framing effects:
Distribution of effect sizes, individual differences, and independence of types of
effects. Organizational Behavior and Human Decision Processes, 88(1), 411-429.
doi:10.1006/obhd.2001.2983

Lobello, S. G. (1996). Review of the Hogan Personality Inventory (Revised). In J. C. Impara &
J. C. Conoley (Eds.), The Supplement to the Twelfth Mental Measurements Yearbook.
Lincoln: The University of Nebraska Press.

Locke, E. A., & Latham, G. P. (1990). A theory of goal setting and task performance.
Englewood Cliffs, NJ: Prentice Hall.

Lovallo, D., & Sibony, O. (2010). The case for behavioral strategy. Mckinsey quarterly.
Retrieved from
http://www.mckinsey.com/insights/strategy/the_case_for_behavioral_strategy

Marsden, K. (1998). The relationship between self-efficacy, intrinsic motivation, and


performance following negative feedback: A study of collegiate
swimmers. Dissertation Abstracts International, 58.

McGhee, R. L., Ehrler, D. J., Buckhalt, J. A., & Phillips, C. (2012). The relation between five-
factor personality traits and risk-taking behavior in preadolescents. Psychology, 3,
558-561. doi:10.4236/psych.2012.38083

McGraw, A. P., Larsen, J. T., Kahneman, D., & Schkade, D. (2010). Comparing gains and
losses. Psychological Science, 21, 14381445.

93

REFERENCES

McIntyre, D. A. (2012). The worst business decisions of all time. Retrieved from
http://247wallst.com/2012/10/17/the-worst-business-decisions-of-all-time/.

McNaught, J. E. (2013). How Baby Boomers experienced leaders use intuition in decision-
making. Dissertation Abstracts International Section A, 73.

McPhail, S. M. (2007). Alternative validation strategies. San Francisco: Jossey-Bass.

Milkman, K. L., Chugh, D., & Bazerman, M. H. (2009). How can decision-making be
improved? Perspectives on Psychological Science, 4, 379-383. doi:10.1111/j.1745-
6924.2009.01142.x

Mintzberg, H., & Westley, F. (2001). Decision-making: It's not what you think. MIT Sloan
Management Review, 42, 89-93.

Mol, S. T., Born, M. P., Willemsen, M. E., & Van Der Molen, H. T. (2005). Predicting Expatriate
Job Performance for Selection Purposes A Quantitative Review. Journal of Cross-
Cultural Psychology, 36, 590-620.

Nutt, P. C. (1999). Surprising but true: Half the decisions in organizations fail. The Academy
of Management Executive, 13, 75-90.

Nutt, P. C. (2004). Expanding the search for alternatives during strategic decision-
making. The Academy of Management Executive, 18, 13-28.

O'Connell, G., Christakou, A., Haffey, A. T., & Chakrabarti, B. (2013). The role of empathy in
choosing rewards from another's perspective. Frontiers in Human Neuroscience, 7.
doi:10.3389/fnhum.2013.00174

Organ, D. W., & Ryan, K. (1995). A meta-analytic review of attitudinal and dispositional
predictors of organizational citizenship behavior. Personnel Psychology, 48, 775-802.

Pachur, T., & Hertwig, R. (2006). On the psychology of the recognition heuristic: retrieval
primacy as a key determinant of its use. Journal of Experimental Psychology:
Learning, Memory, and Cognition, 32, 983.

Parker, A. M., & Fischhoff, B. (2005). Decision-making competence: External validation


through an individual-differences approach. Journal of Behavioral Decision-making,
18, 127.

Payne, J. W., Bettman, J. R., & Johnson, E. J. (1993). The adaptive decision maker.
Cambridge University Press.

Pellegrino, K. (1996). Strategic thinking ability: Cognitive and personality


effects. Dissertation Abstracts International Section A: Humanities and Social
Sciences, 57.

94

REFERENCES

Peeters, M. A., Van Tuijl, H. F., Rutte, C. G., & Reymen, I. M. (2006). Personality and team
performance: a meta-analysis. European Journal of Personality, 20, 377-396.

Rieskamp, J., & Otto, P. E. (2006). SSL: a theory of how people learn to select
strategies. Journal of Experimental Psychology: General, 135, 207.

Rozin, P., & Royzman, E. B. (2001). Negativity bias, negativity dominance, and contagion.
Personality and Social Psychology Review, 5, 296320.

Rubinstein, A. (1998). Modeling bounded rationality (Vol. 1). MIT press.

Russ, F. A., McNeilly, K. M., & Comer, J. M. (1996). Leadership, decision-making and
performance of sales managers: A multi-level approach. Journal of Personal Selling
and Sales Management, 16, 1-15.

Russell, M., & Karol, D. (2002). 16PF Fifth Edition with updated norms administrators
manual. Champaign, IL: Institute for Personality and Ability Testing, Inc.

Safi, A., & Burrell, D. N. (2007). Developing Advanced Decision-making Skills in International
Leaders and Managers. Vikalpa, 32, 1-8.

Salmoni, A. W., Schmidt, R. A., & Walter, C. B. (1984). Knowledge of results and motor
learning: a review and critical reappraisal. Psychological Bulletin, 95, 355.

Saucier, G. (1994). Mini-markers: A brief version of Goldbergs unipolar Big-Five markers.


Journal of Personality Assessment, 63, 506-516.

Schmidt, J. B., & Calantone, R. J. (2002). Escalation of commitment during new product
development. Journal of Academy of Marketing Science, 103-118.

Shah, A. K., & Oppenheimer, D. M. (2008). Heuristics made easy: An effort-reduction


framework. Psychological Bulletin, 207-222.

Shrauger, J. S., & Rosenberg, S. E. (1970). Self-esteem and the effects of success and
failure feedback on performance1. Journal of Personality, 38, 404-417.

Simon, H. A. (1979). Rational decision-making in business organizations. The American


Economic Review, 69, 493-513.

Smither, J. W., London, M., & Reilly, R. R. (2005). Does performance improve following
multisource feedback? A theoretical model, meta-analysis, and review of empirical
findings. Personnel Psychology, 58, 33-66. doi:10.1111/j.1744-6570.2005.514_1.x

Snook, B., Taylor, P. J., & Bennell, C. (2004). Geographic profiling: The fast, frugal, and
accurate way. Applied Cognitive Psychology, 18, 105-121.

Snow, C. P. (1956). The two cultures. New Statesman, 6, 413-414.

95

REFERENCES

Snow, C. P. (1959). The two cultures and the scientific revolution: The rede lecture, 1959.
University Press.

Soane, E., Dewberry, C., & Narendran, S. (2010). The role of perceived costs and perceived
benefits in the relationship between personality and risk-related choices. Journal of
Risk Research, 13, 303318.

Society for Industrial and Organizational Psychology (2003). Principles for the validation and
use of personnel selection procedures (4th ed.). Bowling Green, OH: Author.

Spence, A., & Townsend, E. (2008). Spontaneous evaluations: Similarities and differences
between the affect heuristic and implicit attitudes. Cognition and Emotion, 22, 83-
93.

Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning: Implications for
the rationality debate? Behavioral and Brain Sciences, 23, 645726.

Steers, R. M. (1975). Task-goal attributes, n achievement, and supervisory


performance. Organizational Behavior and Human Performance, 13, 392-403.

Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Psychological science can improve
diagnostic decisions. Psychological Science in the Public Interest, 1, 1-26.

Tang, Y., Qian, C., Chen, G., & Shen, R. (2014). How CEO hubris affects corporate social
(ir)responsibility. Strategic Management Journal, Forthcoming.

Tellegen, A., & Waller, N. G. (2008). Exploring personality through test construction:
Development of the Multidimensional Personality Questionnaire. The SAGE
handbook of personality theory and assessment, 2, 261-292.

Tetlock, P. E. (2005). Expert political judgment. Princeton, NJ: Princeton University Press.

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and
biases. Science, 185, 1124-1131.

Weber, E. U., Blais, A., & Betz, N. E. (2002). A domain-specific risk-attitude scale: Measuring
risk perceptions and risk behaviors. Journal of Behavioral Decision-making, 15, 263-
290. doi:10.1002/bdm.414

Weir, S. (2005). Historys worst decisions and the people who made them. New York, NY:
EYE Quarto, Inc.

Wiggins, J. S. (Ed.). (1996). The five-factor model of personality. New York: Guilford.

Wilson, G. D., Barrett, P. T., & Gray, J. A. (1989). Human reactions to reward and punishment:
A questionnaire examination of Gray's personality theory. British Journal of
Psychology, 80, 509-515.

96

REFERENCES

Wilson, S. M. (2010). Decision-making by school psychologists: Use of the


representativeness heuristic and importance of assessment data in determination of
special education eligibility (Doctoral dissertation). Retrieved from UMI. (3420304).

Zajonc, R. B. (1980). Feeling and thinking: Preferences need no inferences. American


Psychologist, 151-175.

Zenger, J., & Folkman, J. (2014, March 10). Do you need to lighten up or toughen up? [Web
log message]. Retrieved from https://hbr.org/2014/03/do-you-need-to-lighten-up-or-
toughen-up

Zuckerman, M. (2007). Sensation seeking and risky behavior. Washington, DC: American
Psychological Association.

Zhu, D. H., & Chen, G. (2014). CEO narcissism and the impact of interlocks on corporate
strategy. Administrative Science Quarterly, Forthcoming.

97

APPENDIX A: SAMPLE HOGAN JUDGMENT REPORT

APPENDIX A: Sample Hogan Judgment Report

98

APPENDIX A: SAMPLE HOGAN JUDGMENT REPORT

APPENDIX A: Sample Hogan Judgment Report (Continued)

99

APPENDIX A: SAMPLE HOGAN JUDGMENT REPORT

APPENDIX A: Sample Hogan Judgment Report (Continued)

100

APPENDIX A: SAMPLE HOGAN JUDGMENT REPORT

APPENDIX A: Sample Hogan Judgment Report (Continued)

101

APPENDIX A: SAMPLE HOGAN JUDGMENT REPORT

APPENDIX A: Sample Hogan Judgment Report (Continued)

102

APPENDIX A: SAMPLE HOGAN JUDGMENT REPORT

APPENDIX A: Sample Hogan Judgment Report (Continued)

103

APPENDIX A: SAMPLE HOGAN JUDGMENT REPORT

APPENDIX A: Sample Hogan Judgment Report (Continued)

104

APPENDIX A: SAMPLE HOGAN JUDGMENT REPORT

APPENDIX A: Sample Hogan Judgment Report (Continued)

105

APPENDIX A: SAMPLE HOGAN JUDGMENT REPORT

APPENDIX A: Sample Hogan Judgment Report (Continued)

106

APPENDIX B: COMPLETE CORRELATION MATRICES

APPENDIX B: Complete Correlation Matrices

Table B.1
Correlations Between Judgment Scales and HPI Scales & Subscales
1 2 3 4 5 6 7 8 9
Adjustment .08 .01 -.03 .76** -.02 .08 .56** .02 .08
Empathy -.10 -.06 .05 .67** .10 -.04 .50** -.10 .02
Not Anxious .23* .11 .09 .46** .01 .07 .37** -.07 -.04
No Guilt .10 -.01 -.14 .48** -.18 .22* .37** .17 .16
Calmness .13 .02 -.17 .49** -.11 .14 .34** -.04 -.09
Even Tempered .03 -.08 -.04 .60** .11 -.05 .44** -.07 .01
No Complaints .04 .04 .06 .50** -.10 .09 .34** .01 .05
Trusting .10 .20 .11 .29** .01 .14 .30** .09 .26*
Good Attachment -.02 -.08 -.06 .33** .06 -.12 .18 .04 .02
Ambition .34** .66** .01 .29** -.58** .49** .18 .05 .13
Competitive .40** .38** -.17 .02 -.51** .36** -.06 .19 .13
Self Confidence .25* .39** .00 .26** -.31** .29** .20* -.04 .12
Accomplishment .14 .31** .10 .37** -.30** .21* .20* -.08 .02
Leadership .30** .57** -.08 .05 -.64** .47** -.02 .09 .11
Identity .11 .10 -.01 .40** -.16 .29** .37** .12 .12
No Social Anxiety .14 .63** .19 .19 -.23* .24* .17 -.09 .02
Sociability .39** .46** -.02 -.12 -.39** .26* -.14 .09 .10
Likes Parties .13 .30** .07 -.03 -.18 .13 -.05 -.16 -.13
Likes Crowds .11 .22* .09 -.06 -.21* .22* .00 .20* .08
Experience Seeking .60** .38** -.27** -.10 -.41** .34** -.08 .02 .25*
Exhibitionistic .25* .29** -.02 -.16 -.23* .02 -.25* .04 .07
Entertaining .05 .20 .10 .01 -.13 .07 -.02 .16 -.01
Interpersonal Sensitivity -.04 .25* .05 .43** -.07 .12 .31** -.17 -.03
Easy to Live With -.07 .07 .03 .51** .04 -.10 .28** -.26** -.11
Sensitivity -.09 .02 -.03 .18 -.05 .08 .13 -.13 -.10
Caring -.11 .05 -.04 .28** -.04 .05 .18 .00 .03
Likes People .11 .45** .10 .14 -.25* .27** .13 -.08 .03
No Hostility -.16 -.01 -.02 .47** .21* -.12 .36** -.13 -.07
Prudence -.29** -.03 .09 .48** .11 -.09 .34** -.04 -.11
Moralistic -.13 -.07 .00 .34** .02 .00 .23* -.18 -.25*
Mastery -.04 -.02 .00 .04 -.16 .10 .00 .05 -.08
Virtuous -.07 .04 .11 .51** .03 .08 .41** -.10 -.16
Not Autonomous -.07 .09 -.04 -.19 .02 -.11 -.18 -.09 -.06
Not Spontaneous -.23* -.07 .12 .05 .17 -.05 .12 .11 .03
Impulse Control -.42** -.17 .05 .26* .20 -.25* .11 .11 -.02
Avoids Trouble .10 .10 -.02 .42** .02 -.03 .27** -.01 .17
Inquisitive .57** .19 -.47** .03 -.34** .31** .00 .19 .22*
Science Ability .45** .16 -.55** -.01 -.34** .32** -.02 .13 .19
Curiosity .45** .02 -.50** .02 -.22* .15 -.04 .16 .13
Thrill Seeking .41** .27** -.05 -.11 -.27** .22* -.09 -.01 .02
Intellectual Games .13 .04 -.20 -.02 -.08 .04 -.06 .33** .22*
Generates Ideas .29** .30** -.05 .17 -.39** .20* .01 .13 .10
Culture .24* -.13 -.26** .06 .11 .09 .14 -.01 .09
Learning Approach .31** .20* -.33** -.08 -.27** .13 -.12 .32** .26**
Education .33** .15 -.18 -.01 -.20* .12 -.02 .26* .30**
Math Ability .50** .25* -.38** -.09 -.42** .20* -.19 .28** .25*
Good Memory .06 .13 -.19 -.04 -.15 .04 -.06 .17 .09
Reading -.01 .03 -.14 -.08 .01 -.02 -.08 .16 .07
Note. N = 98; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

107

APPENDIX B: COMPLETE CORRELATION MATRICES

Table B.2
Correlations Between Judgment Scales and HDS Scales & Subscales
1 2 3 4 5 6 7 8 9
Excitable -.17 -.31** -.13 -.47** .20 -.30** -.40** .02 -.03
Volatile -.01 -.13 -.14 -.52** -.03 -.11 -.43** .08 .04
Easily Disappointed -.04 -.19 -.10 -.34** .10 -.30** -.38** -.04 .03
No Direction -.35** -.36** -.03 -.14 .39** -.23* -.02 .02 -.15
Skeptical -.03 -.26* .10 -.45** .08 -.25* -.41** .02 -.01
Cynical .02 -.09 -.02 -.38** .09 -.11 -.26* -.04 -.09
Mistrusting .08 -.22* .06 -.35** .06 -.35** -.42** .04 .07
Grudges -.14 -.25* .14 -.30** .05 -.12 -.25* .03 .00
Cautious -.35** -.62** -.06 -.17 .44** -.42** -.18 .09 .04
Avoidant -.28** -.62** .00 -.06 .48** -.39** -.04 .09 .04
Fearful -.39** -.59** -.01 -.27* .43** -.40** -.23* .18 .07
Unassertive -.15 -.27* -.13 -.06 .13 -.21* -.15 -.05 -.02
Reserved -.21 -.31** -.18 -.05 .26* -.29** -.06 .07 .09
Introverted -.27* -.35** -.18 -.07 .29** -.35** -.13 .04 .06
Unsocial -.20 -.24* -.14 -.12 .30** -.34** -.13 .15 .17
Tough -.04 -.18 -.12 .10 .03 .01 .11 -.06 -.05
Leisurely -.16 -.24* -.14 -.24* .12 -.32** -.32** -.04 -.10
Passive Aggressive -.11 -.24* .05 -.04 .17 -.28** -.14 .01 -.05
Unappreciated -.06 -.12 -.11 -.36* .03 -.22* -.36** -.16 -.02
Irritated -.16 -.14 -.28** -.13 .04 -.17 -.20 .06 -.14
Bold .17 .36** -.11 .17 -.55** .24* -.06 -.19 -.20
Entitled .08 .22* -.17 .11 -.24* .19 .04 -.19 -.06
Overconfidence .04 .21 .00 .29** -.41** .14 .02 -.16 -.22*
Fantasized Talent .25* .41** -.08 .00 -.59** .22* -.20 -.09 -.17
Mischievous .42** .19 -.06 -.15 -.32** .09 -.20 -.05 -.14
Risky .47** .17 -.10 -.16 -.24* .14 -.14 -.06 -.03
Impulsive .34** .19 -.19 -.10 -.29** .13 -.15 -.03 .03
Manipulative .11 .09 .16 -.08 -.21 -.08 -.19 -.01 -.35**
Colorful .47** .44** -.20 -.14 -.39** .40** -.06 .16 -.12
Public Confidence .35** .54** -.08 -.03 -.38** .37** .01 .06 -.21
Distractible .36** .15 -.17 -.23* -.18 .28** -.07 .21 -.02
Self-Display .33** .26* -.19 -.06 -.28* .22* -.09 .09 -.02
Imaginative .27* .19 -.29** -.07 -.43** .15 -.23* -.04 -.12
Eccentric .23* -.01 -.27* -.26* -.05 .10 -.15 .03 .09
Special Sensitivity .18 .22* -.13 .05 -.44** .15 -.14 -.12 -.29**
Creative Thinking .29** .18 -.25* -.08 -.36** .08 -.24* -.03 .01
Diligent -.13 .00 -.06 -.14 -.10 -.15 -.27* -.11 -.11
Standards .00 .09 -.12 -.06 -.30** -.02 -.24* -.06 -.06
Perfectionistic -.15 -.04 -.14 -.28** -.01 -.22* -.36** -.09 -.11
Organized -.12 -.02 .11 .05 .02 -.07 .00 -.09 -.07
Dutiful -.38** -.26* .16 -.09 .16 -.34** -.22* -.03 -.17
Indecisive -.41** -.27* .07 .00 .12 -.18 -.09 -.03 -.10
Ingratiating -.16 -.17 .08 -.17 .14 -.31** -.26* -.02 -.06
Conforming -.33** -.17 .22* -.05 .12 -.30** -.17 -.01 -.23*
Note. N = 85; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

108

APPENDIX B: COMPLETE CORRELATION MATRICES

Table B.3
Correlations Between Judgment Scales and CPI Scales
1 2 3 4 5 6 7 8 9
Dominance .26** .69** -.03 .02 -.53** .42** -.02 -.06 .00
Capacity for Status .36** .55** -.14* .11 -.35** .38** .12 .02 .13
Sociability .31** .51** .01 .07 -.34** .32** .07 -.03 .03
Social Presence .39** .38** -.11 .07 -.25** .25** .06 -.01 .09
Self-Acceptance .41** .67** -.06 .01 -.51** .43** .00 .06 .11
Independence .35** .55** -.10 .22** -.36** .44** .22** -.04 .06
Empathy .32** .37** -.06 .15* -.21** .26** .15* .03 .03
Responsibility -.02 .17* -.13 .27** -.07 .27** .29** .13 .10
Socialization -.14 -.06 -.03 .37** .04 .03 .27** .09 .01
Self-Control -.24** -.09 -.01 .47** .17* .15* .49** .01 -.04
Good Impression -.14 .09 -.06 .42** -.01 .17* .37** -.08 -.04
Communality .11 .16* -.09 .16* -.18* .25** .15* .06 .06
Well-Being .07 .14 -.11 .53** -.07 .33** .51** -.02 -.03
Tolerance .01 .04 -.06 .35** .14 .23** .44** .12 .09
Achievement via Conformance .01 .34** -.11 .32** -.25** .28** .25** .07 .06
Achievement via Independence .24** .27** -.29** .17* -.06 .31** .26** .11 .18*
Intellectual Efficiency .34** .36** -.29** .15* -.15* .41** .26** .16* .19**
Psychological-Mindedness .19** .23** -.28** .23** -.10 .28** .25** .07 .08
Flexibility .24** .04 -.16* .12 .07 .12 .19** .05 .01
Femininity/Masculinity -.35** -.24** .24** -.11 .29** -.26** -.09 .05 -.08
Externality/Internality -.32** -.65** .01 .11 .56** -.30** .19** .04 -.04
Norm-Doubting/Norm-Favoring -.11 .10 -.03 .15* -.18* .05 .02 .05 .03
Ego-Integration .15* .23** -.21** .39** -.03 .34** .44** .08 .10
Note. N = 195; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

109

APPENDIX B: COMPLETE CORRELATION MATRICES

Table B.4
Correlations Between Judgment Scales and NEO PI-R Scales & Facets
1 2 3 4 5 6 7 8 9
Neuroticism -.12 -.29** .06 -.49** .15* -.34** -.44** .12 -.04
Anxiety -.19** -.21** .03 -.38** .12 -.29** -.35** .13 -.06
Angry Hostility .01 -.08 .09 -.60** -.05 -.11 -.47** .10 .03
Depression -.08 -.27** .03 -.37** .15* -.27** -.32** .09 -.08
Self-Consciousness -.27** -.42** .04 -.26** .22** -.41** -.30** .07 -.08
Impulsiveness .08 -.09 .03 -.34** .02 -.21** -.33** .05 -.01
Vulnerability -.10 -.29** .08 -.31** .25** -.28** -.23** .12 .03
Extraversion .21** .44** .17* -.02 -.36** .23** -.07 -.03 .06
Warmth -.06 .19* .20** .12 -.10 .06 .05 -.06 -.01
Gregariousness .08 .27** .21** .03 -.14 .16* .05 -.05 -.03
Assertiveness .18* .67** .13 -.03 -.52** .32** -.12 -.03 .01
Activity .15* .36** .02 -.12 -.33** .20** -.13 .11 .14
Excitement-Seeking .33** .11 -.01 -.19** -.23** .07 -.21** -.06 .10
Positive Emotions .12 .13 .11 .10 -.07 .08 .07 -.02 .04
Openness .31** .11 -.33** -.19** -.11 .13 -.11 .07 .12
Fantasy .26** .10 -.22** -.18* -.14 .04 -.17* .11 .10
Aesthetics .16* .01 -.17* -.16* .00 .08 -.06 .03 .00
Feelings .05 .06 .03 -.30** -.07 -.02 -.24** .08 .08
Actions .39** .14 -.17* .03 -.12 .24** .10 .00 .01
Ideas .24** .19** -.44** -.18* -.24** .14 -.19* .06 .19**
Values .12 -.06 -.27** .05 .13 .04 .13 .00 .07
Agreeableness -.29** -.17* .04 .28** .29** -.06 .30** -.13 -.09
Trust -.10 .11 .03 .33** .08 .10 .31** -.13 -.07
Straightforwardness -.27** -.12 .00 .13 .23** .05 .26** -.09 -.01
Altruism -.23** .01 .06 .18* .01 -.03 .09 -.04 -.07
Compliance -.22** -.17* .01 .39** .25** -.13 .30** -.13 -.14
Modesty -.23** -.36** -.02 .00 .35** -.14 .12 -.01 -.04
Tender-Mindedness -.11 -.15* .09 .05 .20** -.10 .07 -.09 -.05
Conscientiousness -.06 .27** -.03 .11 -.33** .18* -.01 .03 .03
Competence .02 .28** -.09 .18* -.31** .20** .05 .02 .10
Order -.04 .09 .12 -.07 -.14 .10 -.05 .02 -.01
Dutifulness -.20** .14 .05 .04 -.20** .01 -.08 -.02 .09
Achievement Striving .14 .40** -.15* -.02 -.45** .20** -.15* .01 -.02
Self-Discipline -.02 .29** .04 .16* -.32** .24** .08 .03 .02
Deliberation -.20** -.04 -.12 .21** .02 .00 .13 .08 -.01
Note. N = 182; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

Table B.5
Correlations Between Judgment Scales and IPIP Big 5 20-Item Scales
1 2 3 4 5 6 7 8 9
Extraversion .21** .52** .18* -.02 -.34** .26** -.05 -.07 .02
Agreeableness .02 .11 .10 .13 -.08 .16* .13 -.03 .01
Conscientiousness -.11 .18* .00 .00 -.24** .20** .01 -.07 -.03
Emotional Stability .14 .14 -.08 .57** .00 .18* .50** -.16* -.05
Intellect / Imagination .32** .35** -.31** -.12 -.35** .25** -.13 .15* .28**
Note. N = 169; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

110

APPENDIX B: COMPLETE CORRELATION MATRICES

Table B.6
Correlations Between Judgment Scales and 16PF Scales
1 2 3 4 5 6 7 8 9
Warmth .00 .21** .37** .06 -.06 .12 .09 -.11 -.04
Reasoning .12 .09 -.26** -.01 -.04 .14* .05 .28** .41**
Emotional Stability .11 .19** .02 .38** -.15* .31** .37** -.03 .00
Dominance .31** .48** -.08 -.22** -.43** .32** -.17* -.08 .07
Liveliness .12 .15* .12 -.11 -.17* .01 -.15* -.03 .07
Rule-Consciousness -.18* .06 .12 .14* -.07 .06 .10 -.05 -.09
Social-Boldness .12 .59** .20** .00 -.37** .25** -.04 -.12 -.04
Sensitivity -.16* -.06 .25** -.01 .16* -.12 -.02 .02 .03
Vigilance -.04 -.15* -.04 -.20** .01 -.18* -.24** -.04 -.09
Abstractedness .24** .05 -.23** -.20** -.10 -.08 -.25** .03 .07
Privateness .02 -.25** -.25** .04 .14 -.22** -.05 .07 .03
Apprehension -.27** -.27** .03 -.29** .14 -.34** -.35** -.03 -.11
Openness to Change .36** .28** -.24** -.16* -.18* .26** -.05 -.03 .11
Self-Reliance -.11 -.13 -.07 -.06 .07 -.24** -.15* -.09 -.04
Perfectionism -.18* .06 .05 -.08 -.21** .14* -.07 -.02 .05
Tension .00 -.07 -.07 -.28** .00 -.12 -.28** .09 .16*
Note. N = 197; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.


Table B.7
Correlations Between Judgment Scales and TCI Scales
1 2 3 4 5 6 7 8 9
Exploratory Excitability vs. Stoic Rigidity .51** .26** -.14* -.09 -.26** .20** -.08 .02 .09
Impulsiveness vs. Reflection .26** .12 .19** .03 .01 .06 .06 -.08 -.01
Extravagance vs. Reserve .05 .02 .12 -.14* .04 -.03 -.07 .06 .07
Disorderliness vs. Regimentation .24** .16* -.06 -.07 -.22** .00 -.15* .05 .07
Pessimism vs. Optimism -.27** -.27** .03 -.29** .26** -.40** -.31** .11 -.02
Fear of Uncertainty -.50** -.27** .10 -.04 .27** -.27** -.05 .15* -.02
Shyness with Strangers -.27** -.53** -.08 -.12 .37** -.30** -.06 .10 .02
Fatigability and Asthenia -.22** -.21** .12 -.19** .13 -.21** -.18** .02 -.07
Sentimentality -.14* -.05 .21** -.04 .01 -.11 -.09 -.03 .03
Openness to Communication vs. Aloofness .07 .34** .24** -.01 -.22** .26** .05 -.06 -.01
Attachment .04 .22** .17* -.04 -.09 .25** .09 -.04 .00
Dependence -.29** -.11 .20** .17* .19** -.14* .11 .01 -.08
Eagerness of Effort vs. Laziness .06 .28** .00 .05 -.32** .23** .02 -.09 -.07
Work-Hardened vs. Spoiled .20** .24** -.17* .04 -.38** .26** -.02 .02 .05
Ambitious vs. Underachieving .14* .44** -.14* -.04 -.53** .18* -.20** -.02 -.02
Perfectionist vs. Pragmatist .07 .28** -.20** -.06 -.36** .15* -.15* -.04 .00
Responsibility vs. Blaming .22** .24** .03 .29** -.19** .28** .26** .07 .05
Purposefulness vs. Lack of Goals .09 .38** -.11 .28** -.29** .27** .19** -.03 .05
Resourcefulness .33** .48** -.16* .10 -.44** .45** .11 -.01 .06
Self-Acceptance vs. Self-Striving .11 -.02 .04 .30** .16* .24** .43** -.01 .01
Enlightened Second Nature -.07 .13 .02 .28** -.11 .16* .22** -.04 -.02
Social Acceptance vs. Social Intolerance .08 .05 .08 .22** .07 .11 .26** -.01 -.03
Empathy vs. Social Disinterest .04 .10 .05 .08 -.10 .06 .02 -.07 -.04
Helpfulness vs. Unhelpfulness .02 .16* .01 .18* -.05 .20** .23** .10 .04
Compassion vs. Revengefulness .07 .12 .05 .27** -.01 .18* .28** -.09 -.07
Conscience vs. Self-Serving Advantage -.02 .14* .05 .04 -.02 .12 .09 .13 .13
Self-Forgetful vs. Self-Conscious .21** .12 -.10 -.16* -.29** .04 -.26** -.03 -.02
Identification vs. Self-Differentiation .12 .12 -.04 -.07 -.21** .07 -.13 -.15* -.09
Spiritual Acceptance vs. Materialism .05 .09 .30** -.09 -.11 .00 -.10 -.05 .02
Enlightened vs. Objective .00 .05 .34** -.06 -.10 .00 -.08 -.01 -.03
Idealistic vs. Practical -.03 .06 .30** -.05 -.10 -.03 -.10 -.04 -.04
Note. N = 203; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

111

APPENDIX B: COMPLETE CORRELATION MATRICES

Table B.8
Correlations Between Judgment Scales and MPQ Scales
1 2 3 4 5 6 7 8 9
Well-Being .16* .24** -.07 .21** -.32** .16* .05 -.08 .03
Social Potency .27** .62** .06 -.16* -.57** .26** -.25** -.08 -.03
Achievement .18* .23** -.28** -.04 -.36** .17** -.12 .04 .06
Social Closeness .04 .26** .16* .12 -.14* .21** .13 .01 .02
Stress Reaction -.13 -.27** .01 -.51** .18** -.30** -.43** .09 -.02
Aggression -.01 -.12 -.01 -.36** .02 -.02 -.22** .11 .02
Alienation .02 -.07 -.03 -.28** .03 -.06 -.19** -.08 -.07
Control -.29** -.03 .01 .17* .04 .01 .14* .10 .07
Harm Avoidance -.36** -.12 .15* .16* .12 -.12 .10 .14* -.06
Traditionalism -.19** .04 .25** .02 -.17* -.07 -.11 -.09 -.14
Absorption .16* -.05 -.11 -.26** -.11 -.11 -.32** -.06 -.04
Note. N = 205; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

Table B.9
Correlations Between Judgment Scales and 6FPQ Factors & Facets
1 2 3 4 5 6 7 8 9
Extraversion Factor .26** .58** .12 -.04 -.50** .35** -.07 -.03 -.03
Affiliation Facet .08 .34** .19** .06 -.25** .18* .02 -.03 -.03
Dominance Facet .26** .52** -.07 -.04 -.49** .38** -.05 -.02 .00
Exhibition Facet .26** .50** .18* -.09 -.41** .24** -.13 -.03 -.05
Agreeableness Factor -.06 -.10 .06 .52** .19** -.09 .38** -.11 -.10
Abasement Facet .05 -.01 .04 .26** .07 -.09 .14 -.06 -.07
Even-Tempered Facet -.15* -.13 .04 .59** .18* -.05 .46** -.11 -.10
Good-Natured Facet -.04 -.11 .07 .40** .21** -.08 .32** -.11 -.06
Independence Factor .17* -.08 -.19** .07 .02 .07 .10 .03 .11
Autonomy Facet .22** -.02 -.22** -.01 -.05 .02 -.03 .04 .13
Individualism Facet .06 -.15* -.06 .19** .16* .12 .29** .01 .05
Self-Reliance Facet .11 -.01 -.17* -.02 -.07 .02 -.04 .01 .08
Openness to Experience Factor .34** .24** -.30** -.10 -.16* .17* -.07 .05 .10
Change Facet .37** .14 -.12 -.06 -.12 .08 -.07 -.02 .03
Understanding Facet .16* .17* -.30** -.10 -.10 .06 -.12 .09 .11
Breadth of Interest Facet .28** .26** -.26** -.06 -.16* .27** .03 .03 .10
Methodicalness Factor -.29** -.04 .06 .06 -.02 .03 .06 -.06 -.04
Cognitive Structure Facet -.32** -.07 .08 .04 .07 -.03 .05 -.10 -.06
Deliberateness Facet -.23** -.05 -.11 .23** -.03 -.01 .11 .02 -.01
Order Facet -.16* .02 .13 -.07 -.07 .09 -.01 -.06 -.03
Industriousness Factor .01 .20** -.27** .01 -.24** .17* -.01 -.04 .00
Achievement Facet .07 .24** -.11 -.04 -.24** .11 -.08 .02 .01
Endurance Facet .15* .17* -.32** .02 -.26** .25** .04 -.06 .03
Seriousness Facet -.17* .05 -.16* .04 -.05 .01 .00 -.03 -.05
Note. N = 200; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

112

APPENDIX B: COMPLETE CORRELATION MATRICES

Table B.10
Correlations Between Judgment Scales and PRS Scales
1 2 3 4 5 6 7 8 9
BIS Anxiety Scale -.24** -.23** .08 -.30** .16* -.31** -.32** .11 -.04
BAS Reward Responsiveness .06 .08 .12 -.14* -.15* -.02 -.20** .09 -.03
BAS Drive .19** .16* -.07 -.18** -.29** .07 -.26** .01 -.02
BAS Fun Seeking .38** .13 -.02 -.28** -.22** .04 -.29** -.01 .10
HEXACO Sincerity Scale .02 .01 .00 .22** .06 .27** .36** .02 -.03
HEXACO Fairness Scale -.16* .10 .10 .25** -.05 .10 .19** .07 .02
HEXACO Greed Avoidance Scale -.03 -.18* .02 .18* .29** .02 .28** .05 .01
HEXACO Modesty Scale -.22** -.26** .28** .21** .41** -.16* .28** .03 .02
HEXACO Fearfulness Scale -.44** -.18* .18** -.01 .24** -.19** .00 .02 -.10
HEXACO Anxiety Scale -.21** -.12 .02 -.39** .02 -.28** -.42** -.05 -.13
HEXACO Dependence Scale -.10 .00 .06 -.14* .03 -.02 -.11 -.03 .01
HEXACO Sentimentality Scale -.18* -.05 .20** -.06 .01 -.11 -.11 -.02 -.06
HEXACO Expressiveness Scale .10 .37** .16* -.30** -.26** .13 -.24** .05 .05
HEXACO Social Boldness Scale .23** .69** .06 .00 -.51** .39** -.03 -.06 -.05
HEXACO Sociability Scale .09 .24** .15* -.06 -.23** .10 -.11 -.05 .01
HEXACO Liveliness Scale .22** .33** -.02 .11 -.25** .22** .08 -.01 .12
HEXACO Forgiveness Scale .05 .00 .04 .34** .09 .03 .28** -.04 .05
HEXACO Gentleness Scale -.13 -.28** .02 .39** .23** -.16* .28** -.13 -.13
HEXACO Flexibility Scale -.05 -.16* .07 .36** .21** -.10 .30** -.08 -.02
HEXACO Patience Scale -.01 -.05 .01 .54** .06 -.04 .33** -.11 -.03
HEXACO Organization Scale -.08 .12 .23** -.01 -.14* .10 -.01 -.07 -.09
HEXACO Diligence Scale .18* .34** -.21** -.02 -.43** .26** -.09 .00 .04
HEXACO Perfectionism Scale -.12 .02 -.14* -.21** -.11 .00 -.20** .03 -.03
HEXACO Prudence Scale -.16* .06 -.12 .37** -.09 .09 .23** .10 .01
HEXACO Aesthetic Appreciation Scale .14* .05 -.20** -.03 .04 .06 .01 .01 .07
HEXACO Inquisitiveness Scale .26** .13 -.52** -.03 -.14* .08 -.08 .03 .12
HEXACO Creativity Scale .41** .28** -.23** -.17* -.32** .19** -.18* .05 .07
HEXACO Unconventionality Scale .34** .13 -.36** -.20** -.16* .12 -.17* .04 .16*
HEXACO Honesty-Humility Scale -.13 -.12 .14* .29** .25** .07 .38** .05 .01
HEXACO Emotional Stability Scale -.32** -.12 .16* -.21** .11 -.21** -.22** -.03 -.09
HEXACO Extraversion Scale .21** .55** .12 -.10 -.42** .28** -.11 -.02 .04
HEXACO Agreeableness Scale -.04 -.15* .04 .52** .18** -.08 .38** -.11 -.04
HEXACO Conscientiousness Scale -.07 .20** -.05 .03 -.28** .16* -.03 .00 -.04
HEXACO Openness to Experience Scale .38** .19** -.43** -.14* -.19** .15* -.14* .04 .14*
Note. N = 208; BIS = Behavioral Inhibition System; BAS = Behavioral Activation System; 1 = Threat Avoidance vs. Reward Seeking; 2 =
Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 =
Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical
Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level.

113

APPENDIX B: COMPLETE CORRELATION MATRICES

Table B.11
Correlations Between Judgment Scales and MVPI Scales & Themes
1 2 3 4 5 6 7 8 9
Aesthetics .20* -.02 -.16 -.29** .00 .05 -.16 .09 .04
Lifestyle .20* -.04 -.17 -.10 .07 .07 .02 .13 .16
Beliefs .09 -.15 -.14 -.26** .08 .09 -.04 .04 -.06
Occupational Preferences .19 .01 -.18 -.20* .00 -.08 -.18 .02 .00
Aversions -.01 .02 -.17 -.29** -.14 .12 -.20* .10 .09
Preferred Associates .24* .05 .03 -.28** -.05 .02 -.21* .06 .01
Affiliation .19 .28** .03 .18 -.18 .12 .06 -.15 -.02
Lifestyle .01 .26* .02 .15 -.12 .13 .10 -.07 -.02
Beliefs .15 .08 -.04 .15 -.01 -.02 .05 -.19 .24*
Occupational Preferences .20* .22* .04 .26** -.17 .21* .19 -.14 -.07
Aversions .20* .16 .02 -.13 -.16 .00 -.20* -.17 -.17
Preferred Associates .09 .14 .04 .16 -.13 -.03 -.01 .03 .05
Altruistic .00 .11 .00 .16 -.04 -.03 .03 -.18 -.01
Lifestyle .03 .15 .06 .19 -.09 .00 .05 -.21* .03
Beliefs -.03 .01 .03 .18 .02 -.02 .09 -.08 .07
Occupational Preferences -.01 .03 .01 .15 .02 -.14 -.02 -.20* -.09
Aversions -.05 .00 -.10 -.08 -.03 -.05 -.12 -.10 -.01
Preferred Associates .06 .19 .08 .26** -.10 .09 .13 -.02 .07
Commercial .17 .21* -.20* -.09 -.39** .16 -.20* -.04 .02
Lifestyle .22* .23* -.28** .02 -.32** .21* -.07 -.01 .02
Beliefs -.06 -.04 .12 .01 -.19 -.10 -.14 -.05 -.09
Occupational Preferences .20 .14 -.20 -.12 -.28** .11 -.18 .03 .11
Aversions -.11 .15 .04 -.15 -.26** .08 -.19 -.11 -.10
Preferred Associates .13 .13 -.17 -.04 -.20* .13 -.07 -.04 .07
Hedonistic .13 -.04 .02 -.36** -.21* -.12 -.42** .07 .16
Lifestyle .20* -.05 .00 -.20* -.19 .04 -.19 .10 .21*
Beliefs .13 -.06 -.12 -.28** -.18 -.06 -.34** .02 .02
Occupational Preferences .10 .01 .11 -.27** -.18 -.14 -.37** .05 .18
Aversions -.04 -.07 .06 -.22* -.07 -.12 -.26** .08 .14
Preferred Associates .06 .08 .04 -.23* -.20* -.11 -.34** .05 .10
Power .27** .41** -.20* -.12 -.58** .36** -.17 -.05 .01
Lifestyle .26* .40** -.23* -.08 -.53** .32** -.14 -.13 -.09
Beliefs .15 .20* -.16 -.04 -.44** .23* -.11 -.13 -.04
Occupational Preferences .40** .47** -.18 -.05 -.61** .36** -.15 .14 .18
Aversions .00 .22* -.09 -.19 -.28** .19 -.16 -.02 .04
Preferred Associates .19 .15 -.08 -.08 -.24* .17 -.06 -.06 -.04
Recognition .12 .25* -.04 -.30** -.41** -.01 -.45** .12 .15
Lifestyle .07 .22* .01 -.27** -.32** -.11 -.42** .19 .15
Beliefs .16 .12 -.02 -.25* -.22* -.02 -.28** .00 .00
Occupational Preferences .17 .33** -.07 -.09 -.45** .05 -.31** .02 .17
Aversions -.07 .01 .04 -.36** -.18 -.05 -.37** .10 .11
Preferred Associates .16 .25* -.13 .07 -.33** .16 -.08 .11 .13
Scientific .10 .05 -.67** -.04 -.11 .12 -.05 .09 .04
Lifestyle .24* .03 -.48** -.04 -.07 .15 .00 .09 .03
Beliefs -.13 -.14 -.26* .03 .07 -.11 -.01 .03 -.09
Occupational Preferences .25* .13 -.72** .12 -.18 .20* .06 .04 .10
Aversions -.12 .02 -.33** -.27** -.08 -.01 -.25* .12 .07
Preferred Associates .12 .15 -.49** .06 -.18 .20* .03 .01 .04
Note. N = 99; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

114

APPENDIX B: COMPLETE CORRELATION MATRICES

Table B.11 (Continued)


Correlations Between Judgment Scales and MVPI Scales & Themes
1 2 3 4 5 6 7 8 9
Security -.57** -.28** .03 .04 .23* -.26** .01 -.15 -.14
Lifestyle -.38** -.05 -.04 .12 .04 -.03 .07 -.08 -.07
Beliefs -.37** -.24* -.12 .07 .07 -.10 .03 -.12 -.04
Occupational Preferences -.35** -.21* .00 .12 .34** -.28** .08 -.09 -.05
Aversions -.54** -.25* .09 -.10 .23* -.33** -.10 -.13 -.14
Preferred Associates -.28** -.21* .14 -.03 .11 -.19 -.05 -.09 -.15
Tradition -.10 .22* .13 .09 -.15 .17 .09 -.04 .10
Lifestyle -.09 .15 .14 .13 -.08 .13 .13 -.16 .01
Beliefs -.01 .18 .13 .05 -.18 .21* .10 -.03 .01
Occupational Preferences -.18 .12 .12 -.01 -.12 .09 .00 -.03 -.01
Aversions .03 .28** -.03 -.05 -.27** .20 -.10 .05 .07
Preferred Associates -.05 .11 .10 .20 .03 .03 .16 .12 .28**
Note. N = 99; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

Table B.12
Correlations Between Judgment Scales and CISS Interest & Skill Scales
CISS Interest Scales 1 2 3 4 5 6 7 8 9
Influencing .26** .46** -.06 .06 -.35** .18* -.04 -.10 .01
Organizing .01 .18* -.09 .05 -.22** .10 -.01 -.04 .00
Helping -.01 .18* .08 .13 .01 .08 .13 .00 .10
Creating .09 .02 .04 -.11 .03 .00 -.06 .07 .06
Analyzing .16* .08 -.57** .07 -.21** .19* .04 .11 .15*
Producing .18* -.04 -.33** .01 -.10 .15 .03 -.03 -.05
Adventuring .31** .15* -.12 .10 -.20* .15 .07 -.03 .06
CISS Skill Scales 1 2 3 4 5 6 7 8 9
Influencing .28** .47** .01 -.01 -.37** .19* -.08 -.01 .03
Organizing .15 .26** -.07 -.04 -.31** .12 -.11 .08 .12
Helping .04 .25** .19* -.01 -.10 .10 .00 .02 .11
Creating .10 .09 .07 -.12 -.08 -.01 -.15 .05 .11
Analyzing .23** .14 -.46** -.01 -.25** .20** -.03 .21** .23**
Producing .21** .01 -.33** -.07 -.19* .15 -.07 -.03 .04
Adventuring .33** .13 -.14 -.05 -.20** .10 -.07 .02 .15
Note. N = 166; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

115

APPENDIX B: COMPLETE CORRELATION MATRICES

Table B.13
Correlations Between Judgment Scales and JPI-R Scales
1 2 3 4 5 6 7 8 9
Analytical Cluster
Complexity .22** .21** -.25** -.10 -.12 .10 -.08 .20** .22**
Breadth of Interest .35** .27** -.34** -.07 -.25** .22** -.09 .01 .07
Innovation .41** .41** -.30** -.08 -.48** .29** -.14* -.02 .05
Tolerance .31** .13 -.12 .19** .01 .23** .28** .04 .01
Emotional Cluster
Empathy -.14* .02 .13 -.11 -.05 -.07 -.16* -.02 -.05
Anxiety -.15* -.09 .00 -.49** .02 -.26** -.50** .10 -.01
Cooperativeness -.25** -.12 .21** -.147* .05 -.33** -.29** -.04 -.18**
Extroverted Cluster
Sociability .12 .22** .11 .03 -.20** .11 -.03 -.05 -.04
Social Confidence .28** .66** .05 .01 -.49** .37** -.03 -.08 -.04
Energy Level .28** .33** -.21** .03 -.30** .29** .03 -.09 -.03
Opportunistic Cluster
Social Astuteness .16* .23** .02 -.15* -.22** -.07 -.27** -.01 -.02
Risk Taking .57** .30** -.18** -.10 -.32** .24** -.09 -.08 .07
Dependable Cluster
Organization -.23** .06 .08 .03 -.15* .09 -.01 .00 .03
Traditional Values -.15* .04 .32** .08 -.11 .00 .01 -.02 -.15*
Responsibility -.13 .09 .05 .14 -.04 .01 .05 -.04 -.03
Note. N = 207; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

Table B.14
Correlations Between Judgment Scales and HBRI Scales
1 2 3 4 5 6 7 8 9
Tactical Thinking .10 -.01 -.16* .03 .01 .06 .05 .45** .36**
Strategic Thinking .19** .09 -.26** .07 -.10 .15* .07 .36** .41**
Overall Critical Reasoning .15* .03 -.21** .06 -.05 .12 .08 .47** .45**
Note. N = 205; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

Table B.15
Correlations Between Judgment Scales and SPA Cognitive Abilities
1 2 3 4 5 6 7 8 9
Verbal Ability .16* .19** -.08 -.05 -.17* .06 -.11 .16* .08
Mathematical Ability .22** .08 -.25** -.00 -.21** .16* -.03 .17* .20**
Note. N = 211; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4
= Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback &
Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; **
Correlation is significant at .01 level.

116

APPENDIX B: COMPLETE CORRELATION MATRICES

Table B.16
Correlations Between Judgment Scales and BRI Cognitive Content
1 2 3 4 5 6 7 8 9
Reading Cluster .08 .15* -.20** .06 -.10 .05 .00 .15* .13
Music/Drama Cluster .12 .16* .02 -.01 -.07 .07 -.01 .05 .17*
Political/Organizational Cluster .22** .27** -.09 .09 -.14* .20** .08 .07 .14*
Books Cluster .13 .07 -.14 -.02 .06 -.02 .00 .15* .20**
Culture Cluster .20** .15* -.19** .07 -.04 .09 .06 .04 .22**
Computing Cluster .12 .13 -.16* .08 -.08 .03 .02 .21** .25**
Work Cluster .15* .17* -.14* -.06 -.07 .19** .07 .32** .34**
Creative Achievements Cluster .23** .25** -.11 -.13 -.15* .17* -.06 .12 .24**
Note. N = 205; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

117

APPENDIX B: COMPLETE CORRELATION MATRICES

Table B.17
Judgment Scale Correlations with Observer Adjective Ratings
1 2 3 4 5 6 7 8 9
Bashful -.19** -.37** -.09 -.06 .24** -.26** -.06 .03 -.04
Bold .15* .42** .14 -.05 -.29** .28** .00 -.07 -.07
Careless .11 .07 .03 -.06 .00 -.03 -.06 -.09 .00
Cold .01 .09 -.05 -.04 -.20** .02 -.13 -.01 .04
Complex .21** .26** -.31** -.18* -.28** .12 -.22** .10 .07
Cooperative -.13 -.10 -.03 .22** .11 .01 .21** -.04 -.10
Creative .14 .14* -.09 -.07 -.14 -.01 -.14 .04 .05
Deep .13 .15* -.14 -.07 -.11 .11 -.05 .17* .13
Disorganized .16* .04 -.12 .03 .01 -.03 .01 -.22** -.07
Efficient -.13 .09 .04 -.03 -.14 .11 -.02 .17* .03
Energetic .15* .24** .03 -.06 -.20** .18* -.04 -.02 .02
Envious .11 .06 .01 -.17* -.01 -.06 -.15* -.05 -.03
Extraverted .08 .31** .14* -.05 -.20** .13 -.05 -.11 -.04
Fretful -.10 -.12 .01 -.22** .06 -.21** -.24** -.07 -.11
Good-Looking -.09 -.02 .20** .02 .07 .05 .12 .04 .03
Harsh .11 .12 -.02 -.23** -.11 -.03 -.25** -.07 .02
Imaginative .19** .20** -.12 -.04 -.14 .11 -.03 .00 .06
Inefficient .12 -.11 -.12 -.02 .10 -.11 -.04 -.07 -.07
Intellectual .12 .24** -.18* -.02 -.25** .10 -.10 .17* .22**
Jealous .03 .08 .06 -.24** -.05 -.08 -.25** -.07 -.11
Kind -.09 -.03 .07 .19** .09 -.07 .13 -.06 .03
Moody -.02 -.07 .03 -.26** .06 -.15* -.23** .01 -.04
Organized -.14* .01 .11 -.01 -.03 .07 .03 .21** .09
Philosophical .16* .20** -.10 -.03 -.15* .08 -.06 .14* .10
Practical -.04 .04 -.18* .11 -.08 .15* .13 .03 .16*
Quiet -.05 -.27** -.30** .08 .08 -.14* -.01 .03 .01
Relaxed .00 -.08 -.07 .30** .05 .07 .25** -.04 .07
Rude .06 .05 -.03 -.18* -.02 -.03 -.15* -.05 -.04
Shy -.10 -.30** -.12 .00 .15* -.21** -.06 .01 -.02
Sloppy .11 -.03 -.12 -.02 .06 -.09 -.02 -.07 .01
Sympathetic -.16* .01 .12 .04 .06 -.03 .04 -.06 -.05
Systematic -.09 .05 -.14* -.05 -.12 .07 -.06 .15* .13
Talkative -.02 .22** .17* -.10 -.07 .08 -.03 -.19** -.12
Temperamental -.02 .06 .04 -.30** -.03 -.12 -.30** -.13 -.10
Touchy .02 .04 .07 -.23** .02 -.14 -.24** -.11 -.10
Unattractive -.02 .00 -.09 -.08 -.04 -.09 -.14* -.09 -.11
Uncreative -.26** -.20** .13 .01 .19** -.05 .08 -.06 -.12
Unenvious .03 .08 -.08 .19** -.07 .09 .14 .07 .11
Unintellectual -.10 -.21** .17* -.07 .18* -.14 -.04 -.23** -.19**
Unsympathetic .07 .01 -.13 -.12 -.06 .01 -.09 .05 .08
Warm -.09 -.03 .12 .09 .08 .04 .13 -.01 -.05
Withdrawn -.03 -.25** -.08 -.05 .12 -.28** -.17* .08 .04
Note. N = 196; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

118

APPENDIX B: COMPLETE CORRELATION MATRICES

Table B.18
Judgment Scale Correlations with Observer Ratings on Descriptive Phrases
1 2 3 4 5 6 7 8 9
Finds fault with others .12 .11 -.09 -.28** -.06 -.03 -.26** -.02 -.03
Does a thorough job -.12 .03 .03 .02 -.09 .10 .04 .13 .01
Is depressed/blue -.07 -.11 .10 -.28** .08 -.26** -.32** -.04 -.08
Is reserved -.11 -.28** -.19** .09 .15* -.19** .00 .03 -.06
Can be somewhat careless .11 -.01 .04 -.07 .07 -.10 -.08 -.16* -.02
Relaxed/handles stress well .12 .11 -.09 .33** -.17* .20** .23** -.03 .10
Full of energy .22** .23** -.05 -.02 -.22** .22** .01 .03 .08
Starts quarrels with others -.01 -.05 .01 -.24** .09 -.11 -.18* -.10 .02
Can be moody -.02 -.11 .04 -.24** .11 -.18* -.21** -.05 -.03
A reliable worker .05 .08 -.01 .05 -.19** .15* .02 .07 .05
Can be tense -.01 .09 .10 -.32** -.03 -.06 -.25** .04 -.06
Ingenious/deep thinker .08 .24** -.21** -.12 -.28** .13 -.14 .14 .18**
Generates a lot of enthusiasm .07 .29** .13 -.11 -.21** .13 -.10 .00 .01
Has a forgiving nature -.09 -.09 .12 .15* .14* .02 .19** .04 .03
Physically attractive -.07 -.03 .22** .05 .03 .06 .10 .09 .03
Tends to be disorganized .13 .02 -.05 -.01 .06 -.08 -.02 -.17* -.10
Worries a lot -.05 .00 .00 -.27** .02 -.15* -.25** -.07 -.05
Has an active imagination .21** .24** -.11 -.17* -.21** .06 -.19** .08 .07
Tends to be quiet -.06 -.28** -.28** .08 .10 -.15* .00 .08 .01
Generally trusting -.11 -.07 .06 .14 .02 .03 .13 .10 .10
Tends to be lazy -.03 -.23** .07 -.10 .21** -.17* -.04 .02 .03
Gets nervous easily -.13 -.14 .13 -.28** .08 -.28** -.30** -.04 -.08
Emotionally stable/not easily upset .04 .02 -.04 .38** -.03 .16* .33** .04 -.02
Inventive .16* .24** -.13 -.07 -.23** .11 -.10 .11 .11
Has an assertive personality .11 .39** .12 -.09 -.27** .23** -.05 -.06 -.05
Original/comes up with new ideas .15* .22** -.13 -.06 -.25** .10 -.13 .15* .12
Can be cold and aloof .10 .03 -.07 -.12 -.10 -.04 -.17* -.08 -.06
Not good-looking -.03 .01 -.15* -.10 -.05 -.07 -.16* -.09 -.06
Perseveres until the task is finished -.15* .13 .01 .07 -.21** .09 -.01 .09 .08
Values artistic, aesthetic experiences .14* .12 -.07 -.06 -.09 .04 -.08 .03 .11
Sometimes shy/inhibited -.07 -.36** -.16* .02 .24** -.21** .00 .04 .02
Considerate and kind to almost everyone -.17* -.02 .03 .11 .03 -.01 .08 .01 .03
Does things efficiently -.12 .07 .04 .03 -.16* .14* .05 .20** .11
Remains calm in tense situations .08 .09 -.08 .24** -.13 .19** .22** .01 .04
Prefers routine work -.32** -.34** .07 -.08 .29** -.18* .01 .08 -.04
Helpful & unselfish with others -.15* -.05 .05 .18* .03 .03 .16* -.07 -.04
Outgoing/sociable .09 .35** .16* .04 -.20** .24** .08 -.12 -.05
Sometimes rude to others .13 .06 -.08 -.16* -.02 -.05 -.16* -.01 .01
Makes plans and follows through -.06 .15* -.01 .21** -.13 .13 .15* .08 .15*
Likes to reflect/play with ideas .13 .12 -.22** -.06 -.14 .09 -.06 .11 .13
Has few artistic interests -.14 -.13 .11 .11 .08 .03 .15* -.06 -.07
Likes to cooperate with others -.19** -.07 .08 .19** .10 -.02 .18** .06 .00
Easily distracted .03 .05 .06 -.24** .05 -.13 -.21** -.18* -.07
Sophisticated in art, music, literature .08 .10 -.16* -.07 .00 -.02 -.07 .12 .15*
Curious about many different things .22** .15* -.23** -.04 -.13 .06 -.06 .01 .05
Note. N = 196; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 =
Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8
= Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant
at .01 level.

119