APPENDIX A: BEST PRACTICES CHECKLIST FOR EVALUATION OF PILOT

SELECTION/SELECTION PROCEDURES
Job Analysis
Was a job analysis conducted and reported?
Does the analysis conform to accepted professional and legal standards?
Does the analysis report conform to the reporting requirements of the Uniform Guidelines on Employee
Selection Procedures (29 CFR § 1607)?

Predictors
Are the constructs to be assessed by proposed predictors clearly specified (operationally defined)?
Are the constructs to be assessed by proposed predictors based on the job analysis?
What is the reliability of each proposed predictors?
What is the test-retest reliability over what time interval?
What is the parallel form reliability based on what index?
What is the internal consistency based on what index?
What is the construct validity of each proposed predictor?
What evidence is there for convergent/discriminant validity of each proposed predictor?

Criteria
What job performance measures are specified as criteria?
How are the job performance measures (criteria) operationally defined?
What are the performance dimensions are assessed?
When and how are the performance data collected?
What is the reliability of each job performance measure?
For measures based on human observations of performance, what is the degree of inter-rater agreement and
inter-rater reliability for the measures?

Validation
Was a formal, company-specific validation study conducted?
What design was used in that validation study, with how large a sample?
How were the predictors sequenced in terms of stages and composites?
What level of statistical power to detect a relationship of what size was afforded by the study design?

A-1
What protections against Type I errors (false positives) were used?
What was the raw, uncorrected validity coefficient for each predictor at each stage?
What was the raw, uncorrected validity coefficient for any composite scores at each stage?
What corrections were made for direct restriction of range on each predictor and/or composite?
What corrections were made for indirect restriction of range on each predictor and/or composite?
What corrections were made for criterion unreliability?
What was the corrected validity coefficient for each predictor and/or composite at each stage?
By what statistical procedure was the validity coefficient estimated (simple correlation, multiple linear
regression, logistic regression, etc.)?

Decision Rules
How were cut-scores on each predictor and/or composite at each stage determined?
What decision rules were established for making selections (e.g., decisions to allow an applicant to proceed
to the next stage and to make a formal bona fide job offer)?

Fairness
What is the adverse impact ratio associated with each decision point?
On what sample sizes were the adverse impact ratios computed?
What alternatives of equal validity but with lesser adverse impact are available, at what cost?

Utility
What changes in training success rates and/or on-the-job performance were estimated on the basis of the
validation study (or studies)?
What is the monetary value of those changes in training success rates and/or job performance to the
company?

Documentation
Is the development and validation of the pilot selection process documented in accordance with 29 C.F.R.
§ 1607.15?
Are applicant records maintained for statistical purposes with personally identifying information to enable
matching to employment data?
Are applicant flow analyses conducted in accordance with 29 C.F.R. § 1607 and OFCCP guidance?

A-2
Are post-hire simulator, check-ride, and/or other performance data collected and matched to applicant
records?

On-going Evaluation
What periodic assessments are made of the fairness, validity, and utility of the selection process (from initial
application through actual entry-on-duty)? How often?

Other Factors
By what organization were any formal validity studies conducted?
To what degree were the organizations conducting such validity studies independent of the company (air
carrier or training school)?
What are the professional credentials of the individuals conducting such validity studies?

A-3