ApplicationofTextAnalysistoQualityControl

ofHumanResourcesDocuments
ThorD.Osborn

Info.sandia.gov/sysanalysis/
Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin
Corporation, for the U.S. Department of Energys National Nuclear Security Administration under contract DE-AC04-94AL85000. SAND2016-8120 C
MotivationforQualityControlofHumanResourcesDocuments

Thejobdescriptionset(JDS)ofmostlargeorganizationssitsatthenexus
ofmanystrategic,operational,andindividualdecisions
EqualpayforequalworkisprotectedbylawintheU.S.andelsewhere
Confoundedjobswithdifferingpayscalesmaybecontestedas
equivalent,incurringsubstantialrisksandliabilities
Recentlegalandpolicyshiftshaveledtoincreasingaccountabilityfor
organizationalimproprietiesamongtopleadership,regardlessoftheir
directinvolvement
ContinuousimprovementoftheJDSmitigatesriskovertimeandsignals
positiveintent

2
AnalysisOutline

Establishobjectivecriteriafor:
Documentsetdifferentiability
Flaggingconfounded(close)jobpairsforexamination
Determineappropriatetermweightingmethod
Demonstrate
ImpactofaddingpoorlydifferentiatedcontenttotheJDS
JDSadjustmenttoimprovejobfamilyclassificationperformance
JDSadjustmenttoimproveoveralldifferentiability

3
ExperimentalJobDescriptionSet
Major Group Description N
Notionalhospitalsystem(NHS) 11 Management Occupations 32
13 Business and Financial Operations Occupations 24
Representsthejobdescription 15 Computer and Mathematical Occupations 7
setnecessarytooperatea 17 Architecture and Engineering Occupations 1
19 Life, Physical, and Social Science Occupations 9
regionalhospitalsystem 21 Community and Social Service Occupations 5
23 Legal Occupations 2
Notmodeledafteranyspecific 25 Education, Training, and Library Occupations 1
realorganization 27 Arts, Design, Entertainment, Sports, and Media Occupations 5
29 Healthcare Practitioners and Technical Occupations 97
250jobdescriptions 31 Healthcare Support Occupations 9
33 Protective Service Occupations 1
15SOCmajorgroups 35 Food Preparation and Serving Related Occupations 11
Building and Grounds Cleaning and Maintenance
37 6
Occupations
43 Office and Administrative Support Occupations 31
49 Installation, Maintenance, and Repair Occupations 3
51 Production Occupations 4
53 Transportation and Material Moving Occupations 2

4
ExampleofJobDifferentiationintheDesignConceptSpace

7 Jobs 21 Job-Pairs D+ Randomvirtualjobs

producedbysampling
Conceptvectorswith
replacement
Virtualjobpairdistribution
representsrandom
designwithintheDesign
ConceptSpace(DCS)
Realjobpairdistribution
expectedtoexhibit
D-
significantlybetter
separation
TwosampleKStest
indicatesrealandvirtual
jobpairsamplesdrawn
fromdifferentdistributions

5
EvaluationProcessSteps
Main (Base Pool) Process
Initial Data Preparation

6
 EvaluationFocusesonSubmedianSeparationDistribution

CompleteNHSJDSused 250jobs,31125job Differentiabilitygovernedbyseparationofclose

pairs (submedian)jobpairs
Realjobpairsfeaturetighterdistribution more Elevatedtailofrealdistributionresponsiblefor
evenlyseparatedwithinDesignConceptSpace KolmogorovSmirnovonesidedD+>0

7
LiveDemonstration1
Estimatevariabilityofwarningthreshold
ShowsubmedianvirtualjobpairsquareddistancedistributionforJDScorpus
Showbootstrapestimateofthe0.1%quantile(warningthresholdstability)
Flagsbetween56and63jobdescriptions(95%confidenceinterval)

8
TermWeightingMethodSelection

TFIDFinappropriateandunstable
Frequencyinappropriate
LogFreq plausiblebutleastconsistent
Binary method chosen
9
AdditionofPoorlyDifferentiatedJobsConfoundsJDS

Sequential addition of 21 Increase in confounded

Physician job descriptions job-pairs with elevation
in order of maximum D- of tail of real distribution

Gradual decline of whole-

JDS differentiability as
DCS becomes crowded

10
 JobFamilyClassification
Discriminant platform interesting rows: Four(4)jobfamilyclassification
errors
Row Actual SqDist(Actual) Prob(Actual) -Log(Prob) Predicted Prob(Pred) Others
ComplianceDirectorhadbeen
98 29 126.065 0.0001 9.417 * 31 0.9999
improperlyassignedtoSOC
109 29 84.638 0.4614 0.773 * 31 0.5386 MajorGroup11
159 11 131.358 0.4115 0.888 * 13 0.5885 Otherjobdescriptions
194 27 90.006 0.0874 2.437 * 13 0.9126 augmentedwithadditional
detail
Secondclassificationanalysis:
Restated in terms of SOC major group: fourerrorsatleftcorrected,but
Janitorreclassifiedto
JobTitle ActualSOC Code PredictedSOC Code HealthcareSupport
Occupations
OrthopedicAssistant HealthcarePractitionersand HealthcareSupport
29 31
PharmacyTechnician TechnicalOccupations Occupations
The DCS will evolve with
ComplianceDirector ManagementOccupations 11 every change to the JDS
BusinessandFinancial
13
GrantWriter
Arts,Design,Entertainment,
27 OperationsOccupations content, potentially altering
Sports,andMediaOccupations classification of jobs other
than those updated

11
LiveDemonstration2
DiscriminantanalysisofJDScorpusaugmentedwithdocumenttopic
vectorstoshowclassificationofjobsbySOCmajorgroup(jobfamily)

12
 RepairingJobFamilyClassification
JanitorisproperlyfoundinSOCMajor Term
Topic56 (+2.3)
Score Term
Topic32 (+1.8)
Score Term
Topic20 (+1.4)
Score
Group37:BuildingandGrounds walk the employe 0.15605 need 0.13776 equipment 0.1513
climb or balance 0.13598 mechanical part 0.11547 pounds frequent lift 0.1373
CleaningandMaintenanceOccupations replac 0.13078 design 0.11233 direct -0.1314
mechanical part fumes 0.13029 schedul 0.10940 must regular lift 0.1229
GreatesttopicaldivergencesofJanitor hall 0.13001 follow duti personally 0.10864 damag 0.1154
expos to move 0.12412 expos to move 0.10393 maintain 0.1052
positionfromSOCMajorGroup37 distanc vision 0.12026 subordin supervisor 0.10311 water 0.1011
shownatright equipment polishes 0.11440 worker 0.10242 balance and stoop 0.0994
metalwork steam 0.10184 kneel crouch 0.0993
AdjustedJanitorjobdescription: polishes floor clean 0.11440 designed 0.10109 prevent 0.0979
sweeps scrubs waxes 0.11440 plan develops 0.09961 improv 0.0977
Addedreferencestowastebaskets, rugs carpets upholstered 0.11440 fund 0.09950 remov 0.0973
door panel 0.11440 establishment by 0.09887 beds 0.0938
waxes,andpolishes empti and clean 0.11440 perform equipment high school 0.0905
Addedphrasesuchasreplacinglight sills empti wastebaskets 0.11440 fabric 0.09824 diploma or general 0.0905
ashtrays transport trash 0.11440 tool 0.09781 educ degree 0.0905
bulbs dusts furniture 0.11440 review 0.09764
woodwork wash window 0.11440 exist 0.09759
AddedsentenceMustbecomfortable furniture and draperies 0.11440 superintendent 0.09727
usingordinaryhandtoolstomake
minorrepairsandadjustmentsto
buildinginfrastructureandequipment.
Zeroclassificationerrorsafter Topical divergence of a job description from target category may
adjustment suggest appropriate adjustments for improving classification

13
 GeneralImprovementofJDSDifferentiation
Closest six job descriptions before revisions: Ninemostconfoundedjob
Distance Job Title A Job Title B Salary Same descriptionschosenfor
Disparity SOC improvementexample
10.0 Ophthalmic Technician Orthoptist 1.17 Yes
10.2 Cafeteria Attendant Counter Supply Worker 1.00 Yes Beforerevisions:
15.0 Cytotechnologist Histotechnologist 1.00 Yes 87jobpairsbelow
16.5 Cardiopulmonary Technologist Electromyographic Technician 1.33 Yes thresholdseparation
18.9 Ophthalmic Technician Optometric Assistant 1.00 Yes
19.4 Optometric Assistant Orthoptist 1.17 Yes 63jobdescriptionsinvolved
Afterrevisions:
Closest six job descriptions after revisions: 63jobpairsbelow
thresholdseparation
Distance Job Title A Job Title B Salary Same
50jobdescriptionsinvolved
Disparity SOC
20.1 Hospital Chaplain Corporate Lawyer 2.62 No RevisionsmadeusingO*NET
21.4 Chief Cardiopulmonary Infection Control Nurse 1.23 Yes
jobrelatedtaskinformation
Technologist
21.5 Assistant Director of Development Director of Corporate Relations 1.07 Yes
21.6 Dentist Oral & Maxillofacial Surgeon 1.94 Yes
22.8 Chief Cardiopulmonary Chief Dietitian 1.06 Yes
Impacts of improvement efforts are
Technologist readily monitored and visualized
23.0 Geneticist Microbiologist 1.11 Yes

14
ConcludingRemarks
Textanalysisapproachshowspromiseasatoolforimprovingmanagementof
organizationaljobdescriptionsets
Approachguidesimprovementeffortsby:
Exposingpriorclassificationerrors
Showingtopicaldiscrepanciessuggestingpossiblegapsindescriptivecontent
Furtherdevelopmentwillrequireanalysisandimprovementeffortsusingjob
descriptionsetsforsizeablerealorganizations
UserfriendlinessenhancementstotheinterfacemaybeessentialforobtainingHR
subjectmatterexpertengagement
Approachmayreduceriskexposures,improveclarityofjobroles,andreduce
proliferationofsubstantiallyidenticaljobsacrosslargeorganizations

15