Quality Assurance Staffing Impacts in Military Aircraft Maintenance Units OK

Journal of Quality in Maintenance Engineering
Quality assurance staffing impacts in military aircraft maintenance units

Terry D. Moore Alan W. Johnson Michael T. Rehg Michael J. Hicks
Article information:
To cite this document:
Terry D. Moore Alan W. Johnson Michael T. Rehg Michael J. Hicks, (2007),"Quality assurance staffing
impacts in military aircraft maintenance units", Journal of Quality in Maintenance Engineering, Vol. 13 Iss 1
pp. 33 - 48
Permanent link to this document:
http://dx.doi.org/10.1108/13552510710735104
Downloaded on: 20 October 2014, At: 11:11 (PT)
Downloaded by Ms Maria Makrygianni At 11:11 20 October 2014 (PT)
References: this document contains references to 16 other documents.

To copy this document: permissions@emeraldinsight.com
The fulltext of this document has been downloaded 790 times since 2007*
Users who downloaded this article also downloaded:
Ahmed A.R. Al#Nakeeb, Trefor Williams, Peter Hibberd, Stuart Gronow, (1998),"Measuring the
effectiveness of quality assurance systems in the construction industry", Property Management, Vol. 16 Iss
4 pp. 222-228
Access to this document was granted through an Emerald subscription provided by

Token:Reviewer:95CDEC49-D372-418B-8719-029336EB15F4:
For Authors
If you would like to write for this, or any other Emerald publication, then please use our Emerald for
Authors service information about how to choose which publication to write for and submission guidelines
are available for all. Please visit www.emeraldinsight.com/authors for more information.
About Emerald www.emeraldinsight.com
Emerald is a global publisher linking research and practice to the benefit of society. The company
manages a portfolio of more than 290 journals and over 2,350 books and book series volumes, as well as
providing an extensive range of online products and additional customer resources and services.
Emerald is both COUNTER 4 and TRANSFER compliant. The organization is a partner of the Committee
on Publication Ethics (COPE) and also works with Portico and the LOCKSS initiative for digital archive
preservation.
*Related content and download information correct at time of download.

The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1355-2511.htm
METHODOLOGY AND THEORY Quality

assurance
Quality assurance staffing staffing
impacts in military aircraft
33
maintenance units
Terry D. Moore
Headquarters, Air Force Materiel Command, Wright-Patterson Air Force Base,
Ohio, USA
Alan W. Johnson
Department of Operational Sciences, Air Force Institute of Technology,

Wright-Patterson Air Force Base, Ohio, USA, and
Michael T. Rehg and Michael J. Hicks
Department of Systems and Engineering Management,
Air Force Institute of Technology, Wright-Patterson Air Force Base, Ohio, USA
Abstract
Purpose – This paper summarizes our research into the impact that current Air Force quality
assurance staffing practices have on key unit performance metrics.
Design/methodology/approach – Interviews and Delphi surveys culminated in the development
of a quality assurance staffing effectiveness matrix. The matrix was used to calculate historical
quality assurance staffing effectiveness at 16 Air Force combat aircraft units. Effectiveness scores
were then regressed with unit historical data for 25 metrics.
Findings – Nine metrics were deemed statistically significant, including break rates, cannibalization
rates, flying schedule effectiveness rates, key task list pass rates, maintenance scheduling
effectiveness rates, quality verification inspection pass rates, repeat rates, dropped objects counts and
safety/technical violations counts. An example benefit-cost analysis for changes in quality assurance
staffing effectiveness presents compelling evidence for maintenance managers to carefully weigh
decisions to leave quality assurance personnel slots empty, or to assign personnel possessing other
than authorized credentials.
Originality/value – Maintenance managers can use this method to help determine personnel
assignment strategies for improving quality assurance unit performance with respect to similar
metrics, and other managers could adopt this general approach to more effectively link personnel
resources to their organization’s key performance indicators.
Keywords Quality assurance, Maintenance, Personnel administration, Performance measures,
Aircraft industry, United States of America
Paper type Research paper
Journal of Quality in Maintenance

Engineering
Vol. 13 No. 1, 2007
q This article is published with permission of the United States Government. pp. 33-48
Emerald Group Publishing Limited
The views expressed are those of the authors and do not represent the official policy or positions 1355-2511
of the United States Air Force, Department of Defense, or the US Government. DOI 10.1108/13552510710735104
JQME 1. Practical implications
13,1 An aircraft maintenance quality assurance unit requires personnel that collectively
possess expertise across a wide variety of functional disciplines. Military aircraft
maintenance managers must provide these experts from within their own organization,
but budget pressures and high operations tempos have left many maintenance units
understaffed – forcing managers to either leave quality assurance slots vacant or to
34 assign people with mismatched skills. Our research introduces a new metric, quality
assurance staffing effectiveness, together with a method for measuring its value. This
metric provides management with the tools necessary to make informed decisions
when assigning quality assurance personnel with high-demand, low-density expertise.
2. Introduction
A typical Air Force combat aircraft wing requires thousands of maintenance
technicians to safely launch, recover, and service aircraft between flights. Wing leaders
rely on a cadre of about 25 experts – the quality assurance unit – to provide daily
maintenance oversight, on-the-spot correction, training, an investigative capacity, and
a mechanism for formal feedback.
The key to an aircraft maintenance quality assurance unit’s effectiveness are the
skills and experience-based qualifications of personnel assigned to the limited
positions. Accordingly, the Air Force performs extensive manpower studies to
determine the authorized quality assurance unit composition that can best perform its
exercise, peacetime, and war duties. The resultant list of authorized positions and
associated required expertise for a particular quality assurance unit is called its Unit
Manning Document (UMD).
Such efforts are important because the Air Force relies heavily on inspection-based
quality assurance. Poorly executed quality assurance can have disastrous
consequences – for example, an F-15 aircraft crashed with loss of plane and pilot
after a mechanic incorrectly installed flight control rods, and another mechanic failed
to catch the miscue (Grier, 1998). In 1996, ValuJet Flight 592 caught fire in-flight and
crashed into the Florida Everglades, killing 110 people. The crash was attributed to
contract maintenance personnel improperly rendering safe and shipping oxygen
cylinders in the cargo hold of the aircraft. A key finding was that ValuJet failed to
adequately oversee the maintenance subcontractor, and this failure was the cause of
the accident (Langewiesche, 1998). Catching such issues and ensuring that critical
inspections are performed correctly are fundamental to the quality assurance function.
There exists a considerable literature on quality assurance staff selection and on
task-skill mismatching. McCormick et al. (1992) discuss the merits of contracting
consultants for skilled management oversight positions, versus using in-house
staffing. Srivastava (2005) shows how a systemic, up-front quality assurance planning
and execution approach enabled two major nuclear power plants to rapidly achieve
goals with 1/3 fewer quality assurance manpower positions. Vessely (1984) presents
guidelines for quality assurance manpower requirements throughout a nuclear power
plant’s production and operations life cycle. Van den Beukel and Molleman (2002)
argue that depending on managerial policies and team social dynamics, employee
multifunctionality can lead to both underutilization of skills and task overload. Zeng
and Zhao (2005) introduce an optimization-based approach for analyzing
role-resolution policies used in current workflow management practice and propose
new policies. However, they assume that all staffing decisions have already been made. Quality
MacDonald et al. (1996) report the results of an internal quality assurance pilot assurance
program at 16 Indonesian hospitals, with the goal of motivating hospital staff to
assume greater responsibility for their quality of service. They provide several staffing
recommendations for success, including comprehensive (versus product-focused)
programs, use of dedicated coaches, and visible leadership from top executives.
A key assumption throughout the literature is that necessary personnel can be 35
readily acquired. Military aircraft maintenance organizations are characterized by the
need to build specialized technical and military skills that require years to develop, in
an environment with a high rate of personnel turnover (approximately five years per
maintenance technician). However, the military cannot generally hire this skilled labor,
but must develop it internally (Dahlman et al., 2002). Furthermore, managers must
provide the experts to fill the quality assurance jobs from within their own
organization—an expert assigned to the quality assurance unit is then unavailable to

perform maintenance duty. A complication is that the organization may already be
undermanned or lack technicians with particular skills or qualifications (known as
specialty codes). Therefore, managers must understand the potential mission impact of
either leaving authorized quality assurance manpower positions unfilled or of
assigning personnel with mismatched specialty codes against these positions.
3. Methodology
Our research was completed in four distinct phases. In phase one, key maintenance
metrics were identified and a staffing effectiveness matrix was constructed. Phase two
consisted of acquiring historical staffing data for 16 Air Force bases and applying the
staffing effectiveness matrix to this data. In phase three, data on the subject aircraft
flying units’ key performance metrics were compiled and analyzed to determine any
relationship with the computed quality assurance staffing effectiveness. Phase four
introduces potential mitigating strategies for use by aircraft maintenance managers.
3.1 Phase one: identifying the metrics and building the effectiveness matrix
We first obtained a list of UMD-authorized manpower positions for all Air Force
combat aircraft flying units. Eliminating administrative skills not directly involved in
quality assurance and combining others led to a final list of 24 different specialty
codes, as shown in Table I. To establish and validate our effectiveness matrix, we used
a Delphi approach, which is a process in which a panel of geographically dispersed
experts deals systematically with a complex task in an iterative manner (The Delphi
Method, 2005). Thirty-four maintenance experts agreed to participate as our Delphi
panel of experts. This panel included at least two members from each Air Force combat
aircraft flying unit, and each possessed a minimum of six years of experience in the
aircraft maintenance field.
The panel assessed the impact that they perceived the quality assurance function
has on a list of 28 candidate wing- and unit-level metrics proposed by us. Our
web-based survey had the respondents rate their perceived quality assurance impact
on the candidate metrics using a six-point scale from Strongly Disagree to Strongly
Agree, and we encouraged them to suggest additional metrics. Twenty-five metrics
were deemed significant by the panel, and are introduced and defined in the Appendix.
Our next survey introduced the quality assurance staffing effectiveness matrix – a tool
JQME
AFSC AFS title
13,1
2A0X1 Avionics Test Station and Computer Journeyman/Craftsman
2A3X0 Tactical Aircraft Superintendent
2A3X1 A10/F15/U2 Avionics Attack Journeyman/Craftsman
2A3X2 A10/F15/U2 Avionics Attack Journeyman/Craftsman
36 2A3X3 Tactical Aircraft Maintenance F-15 Journeyman/Craftsman
2A590 Maintenance Superintendent (Non-Tactical Aircraft)
2A5X1 Aerospace Maintenance Journeyman/Craftsman
2A5X2 Helicopter Maintenance Journeyman/Craftsman
2A5X3 Integrated Avionics Systems/INS Journeyman/Craftsman
2A6X0 Aircraft Systems Manager
2A6X1 Aerospace Propulsion Journeyman/Craftsman
2A6X2 Aerospace Ground Equipment Journeyman/Craftsman
2A6X3 Aircraft Egress Systems Journeyman/Craftsman
2A6X4 Aircraft Fuel Systems Journeyman/Craftsman

2A6X5 Aircraft Hydraulic Systems Journeyman/Craftsman
2A6X6 Aircraft Electrical/Environmental System Journeyman/Craftsman
2A7X3 Aircraft Structural Maintenance Journeyman/Craftsman
2A7X4 Survival Equipment Journeyman/Craftsman
2E1X1 Satellite, Wideband & Telemetry Systems Journeyman/Craftsman
2E2X1 Computer Network S&C Systems journeyman/Craftsman
2M0X1 Missile/SPC Systems Maintenance Journeyman/Craftsman
Table I. 2W0X1 Munition Systems Journeyman/Craftsman
Final list of Air Force 2W1X1 Aircraft Armament Systems Journeyman/Craftsman
Speciality Codes (AFSC) 2W2X1 Nulear Weapons Journeyman/Craftsman
that can measure the average effectiveness of a person with mismatched qualifications
when assigned to quality assurance duty. This survey required two Delphi rounds. In
round 1, a web-based instrument was developed and sent to the Delphi panel. This
instrument asked each Delphi panelist to systematically rate, on a scale of one to five,
how effectively a person possessing a particular specialty code could be expected to
perform the duties and tasks of the 24 quality assurance staffing positions listed in
Table I (Moore, 2005). The panel repeated this ranking process for each of the 24
specialty codes, thus requiring 576 individual effectiveness rankings.
The Delphi panel’s ability to dedicate several hours to complete this survey became
a problem for many of them and thus round 1 took over three months to complete.
Therefore, we developed a filter designed to reduce the panel’s rating effort in the
second Delphi round, by identifying individual effectiveness rankings that could be
deemed as agreed upon (e.g. consistently scored) by the panel. This filter – a coefficient
of variation (CV) discriminator – gave us the ability to compare the variation of two or
more different variables and provided a standardized view across all 576 mean
responses to gain a better understanding of the variability present in the data. Our
hope was that we could reduce the number of individual rankings that would need to
be examined by the panel in round 2. CV discriminators between 0 and 1.0 were
iteratively applied to the mean data responses to illuminate the “fail” responses
(indicating a lack of agreement among the experts) that would need to be addressed by
the panel. After some trial and error, we subjectively settled on a CV factor of 0.29 as
our threshold, which resulted in 529 “fails” to be examined again by the panel. We then
used these “fails” to develop a spreadsheet-based instrument for use in the second Quality
Delphi round. assurance
Panelists were instructed in round 2 to examine the aggregated staffing
effectiveness matrix together with anonymous comments derived from round 1. Of staffing
the 14 respondents in round 2, thirteen modified their round 1 response in varying
degrees. The median data responses from round 2 were then used as a basis to develop
the final quality assurance staffing effectiveness matrix, shown in Figure 1. We 37
determined from e-mail and telephonic responses from the Delphi panel members that
a third survey round would result in no further adjustment to their individual ratings.
3.2 Phase two: determining unit historical quality assurance staffing effectiveness
We obtained monthly personnel assignment data for years 2003 and 2004 from each of
the 16 participating air bases. We then used this data together with the quality
assurance staffing effectiveness matrix to compute an overall monthly historical

quality assurance staffing effectiveness percentage for the assigned personnel for each
base. When assigning an effectiveness rating to each unit’s authorized positions, we
used the following four rules:
(1) If a UMD-authorized quality assurance manpower position was filled with a
person possessing the specialty code called for in the position description, the
position effectiveness was rated 100 percent.
(2) If the person filling a quality assurance position possessed a specialty code
other than that called for in the position description, the appropriate
effectiveness level derived from the quality assurance staffing effectiveness
charts was assigned to that position (e.g. a person with specialty code 2A5X3
assigned to a 2A5X1 quality assurance position would be rated 41 percent
effective, as indicated in Figure 1).
(3) Instances where manpower positions were double-filled (i.e. two persons
possessing specialty code 2A3X3 assigned against one authorized 2A3X3
manpower position), were reconciled by adding an extra position rated as 100
percent effective, and averaging it with the others. The rationale for this was
that, although not authorized for in the UMD, these extra personnel provide an
enhanced capability, thus proper credit should be applied to offset possible
deficiencies in other areas.
(4) All unfilled quality assurance positions were rated as zero percent effective.
After all individual quality assurance manpower positions were assigned staffing
effectiveness ratings, a simple average was computed to determine each quality
assurance unit’s overall staffing effectiveness rating for each month. (Table II depicts
an example calculation.) We then aggregated the monthly staffing effectiveness scores
for all 16 participating quality assurance units into one chart to develop a quality
assurance staffing effectiveness time-series, as shown in Table III.
Phase three: statistical analyses

In phase three we first gathered data covering January 2003 to December 2004 from
each of the 16 participating aircraft maintenance organizations for the 25 metrics
identified by our panel of experts. Using this data together with the calculated staffing
JQME
13,1
38
Figure 1.
Results of Delphi survey,
part 2: quality assurance
effectiveness matrix
Quality
Position ID UMD Assigned Quantity Score %
assurance
1 2A390 Unfilled 0 0 staffing
2 2A373 2A651 1 47
3 2W071 2W071 1 100
4 2A371 2A371 1 100
5 2A373 2A373 1 100 39
6 2A373 2E151 1 14
7 2W171 2W171 1 100
8 2A051D 2A071 1 100
9 2A351B Unfilled 0 0
10 2A353 2A353 1 100
11 2A353 2A353 1 100
12 2A353 2A553 1 80
13 2A353 2A350 1 80
14 2A651A 2A671 1 100

15 2A051 2A071 1 100
16 2A373 2W071 1 12 Table II.
17 2A373 2A373 1 100 Assigned unit quality
18 2W171 2W151 1 100 assurance effectiveness,
Monthly average 74 Pope AFB, January 2004
effectiveness scores for each of the 16 organizations, approximately 2000 total

correlation analyses were performed between the 25 metrics and the respective
organization’s quality assurance staffing effectiveness rating to assess linear
relationships between the variables for various time lags, based on the Pearson
product-moment correlation estimator (Kutner et al., 2005, p. 84). We used the Pearson
product-moment estimator because both staffing effectiveness and each associated
metric are random variables – assigned staffing can change abruptly in a military
unit, depending on world events. Each metric was lagged in monthly increments from
zero (contemporaneous) to four months. (We assume that any effects after four months
would have occurred by chance.) Next, these correlations were aggregated and collated
by metric across all 16 participating organizations. Most of the results were
inconclusive in terms of identifying recurring themes across all 16 organizations and
time lags, as seen in Table IV (Moore, 2005). For example, Abort rates were moderately
negatively correlated (, 2 0.4) with quality assurance staffing effectiveness for some
organizations but weakly to moderately positively correlated for others, which is
counterintuitive (one would expect higher effectiveness to correlate with fewer aborts).
The correlations for some metrics such as the Key Task List Pass rate were consistent
across lags – correlations for most other metrics were not. The correlations did
indicate potential relationships between some metrics and particular organizations –
e.g. twelve organizations exhibited moderate negative correlations between Total
Non-Mission Capable rates and quality assurance staffing effectiveness in at least one
of their flying units during at least one lag period.
We concluded our statistical analysis by first aggregating the metrics data from the
16 participating organizations and then performing 25 respective linear regressions
between quality assurance staffing effectiveness and each metric. The regressions
allowed us to assess the statistical significance of any linear relationships between the
40
13,1
JQME
Table III.
effectiveness
calculated staffing
Quality assurance unit
January February March April May June July August September October November December
score score score score score score score score score score score score
(%) (%) (%) (%) (%) (%) (%) (%) (%) (%) (%) (%)
Barksdale 2003 95 95 95 95 95 95 95 95 95 95 95 95
2004 95 95 95 95 95 95 95 95 95 95 95 95
Beale 2003 66 66 61 63 63 63 59 59 59 56 56 56
2004 56 56 53 56 56 56 56 53 56 56 56 69
Cannon 2003 89 89 89 89 90 90 90 90 87 87 87 87
2004 85 85 85 85 85 85 85 85 86 86 86 86
D-M 2003 80 80 80 84 84 88 88 88 92 80 76 83
2004 79 82 84 84 84 84 84 84 88 81 84 84
Dyess 2003 100 100 100 100 100 100 100 100 100 100 100 100
2004 100 100 100 100 100 96 96 96 95 99 99 99
Ellsworth 2003 78 78 78 78 78 78 79 79 79 79 79 79
2004 79 79 79 78 81 81 81 79 79 79 82 81
Holloman 2003 100 100 100 100 100 100 100 100 100 100 100 100
2004 100 100 100 100 100 100 100 100 100 100 100 100
Langley 2003 85 85 85 85 85 85 85 85 85 85 85 85
2004 85 85 85 85 85 85 84 84 84 87 87 87
Minot 2003 88 88 88 88 88 88 88 88 88 88 88 88
2004 94 94 94 94 94 94 94 94 88 91 91 91
M-H 2003 83 83 80 80 80 79 79 79 79 79 81 81
2004 81 81 81 81 81 81 81 81 81 76 76 76
Nellis 2003 87 87 87 87 87 87 87 87 87 87 87 87
2004 87 87 87 87 87 87 93 93 93 93 93 93
Offutt 2003 64 64 64 64 64 64 64 64 64 68 68 68
2004 71 71 71 75 75 75 75 80 80 80 80 80
Pope 2003
2004 74 74 74 74 80 85 85 78 84 89 89 84
S-J 2003 92 92 92 90 90 90 90 90 90 90 90 90
2004 90 90 91 91 91 91 91 91 92 92 92 92
Shaw 2003 97 97 97 97 97 97 97 97 97 97 97 97
2004 97 97 97 97 97 97 100 100 100 100 100 100
Whiteman 2003 85 85 85 85 85 85 85 85 89 89 89 89
2004 91 91 91 93 91 91 100 91 91 91 91 91
Quality
Number of
Correlation AircraftType/ assurance
Metric direction Correlation strength Units staffing
Abort rate Positive Weak to moderate 8
Negative Weak to moderate 6
Break rate Positive Weak to moderate 11 41
CANN rate Positive Weak to moderate to strong 9
Dropped object counts Positive Weak to moderate 3
Deficiency reports submitted Positive Weak to moderate 4
Detected safety violations Positive Weak to moderate 4

Fix rates Positive Weak to moderate 15
Foreign object damage counts Positive Weak to moderate 3
Flying schedule effectiveness rates Positive Weak to moderate 16
In-flight emergency rates Positive Weak to moderate 9
Key task list pass rates Positive Weak to moderate 4
Mission capable rates Positive Weak to moderate 15
Maintenance scheduling effectiveness rates Positive Weak to moderate 8
Maintenance/operations deviations counts Positive Weak to moderate 10
Personnel evaluation pass rates Positive Weak to moderate 7
Phase key task list pass rates Positive Weak to moderate 4
Quality verification inspection pass rates Positive Weak to moderate 9
Recur rates Positive Weak to moderate 9
Repeat rates Positive Weak to moderate 7
Safety/technical violation counts Positive Weak to moderate 3
Technical data violation counts Positive Weak to moderate to strong 5
Total non-mission capable rates Positive Weak to moderate 6 Table IV.
Negative Weak to moderate 16 Correlation analysis at
T.O. improvements submitted counts Positive Weak to moderate 4 organization level, across
Negative Weak to moderate 6 time lags
JQME variables, and it enabled our use of the regression’s slope values as elasticities for
13,1 specified changes in the staffing effectiveness metric. To compute the regressions, we
arranged the data in a contemporaneous (no-lag) format with each organization and
analyzed the regression output specifically for level of significance and direction of
relationship. Although we had several metrics with respectable R-squared values, the
degree of fit was not our most important consideration. This is because, although the
42 R-squared value is considered as a prime factor when determining usefulness of a
predictive model, we also seek interpretive capacity in the effort to understand how
staffing decisions affect key unit performance indicators. Thus we subjectively
considered R-squares of 0.05 or greater as useful as long as the p-value was significant
(deemed 0.05 or less, see, e.g. Kutner et al. 2005, p. 75). We cannot reasonably expect
that our single quality assurance staffing effectiveness independent variable will
explain all of the variation for any of the 25 candidate dependent metric variables in the
study – there’s just too much process complexity in a typical Air Force combat aircraft
wing. However, the R-squared values do provide useful information nonetheless. One
final concern emerged in our tests for randomness of the regression residuals: in our
data we found five of nine metrics with Durbin-Watson test values that were outside of
the normally acceptable level (Kutner et al., 2005, p. 487), indicating a lack of
randomness. While data transformations could be made, the serial nature of our data
suggests they would not be successful (Oxley, 2000). However, recent studies indicate
that even when heteroscedasticity cannot be eliminated, valid inferences can still be
made (Oxley, 2000). Since we appended our base-level data sets into a single file, we
expect serial correlation. Eight metrics were deemed statistically significant (i.e.
having nonzero regression slope coefficients), as shown in Table V. An additional
metric, Dropped Objects, was significant at the p , 0.15 level. Since it also
demonstrated (in Table IV) a consistently moderately negative correlation with quality
assurance staffing effectiveness at 11 of the 16 bases studied, we included it in Table V
as well.
Phase four: mitigating strategies

Phase four seeks practical usefulness of the results in Table V for aircraft maintenance
managers making quality assurance staffing decisions. While our original intent was
only to develop a tool for measuring the effectiveness of quality assurance staffing
decisions and for understanding the relationships between staffing effectiveness and
Dependent variable Coefficient Adj. R-Square p-value
Break (rate) 2 0.0073% 0.45 ,0.05

Cannibalization (rate) 2 0.0244% 0.61 ,0.01
Flying scheduling effectiveness (rate) 2 0.0038% 0.23 ,0.01
Key task list pass (rate) 2 0.0015% 0.05 ,0.05
Maintenance schedule effectiveness (rate) 2 0.0022% 0.11 ,0.01
QVI Pass (rate) 2 0.0013% 0.18 ,0.01
Table V. Repeat (rate) 2 0.0167% 0.06 ,0.05
Significant regression Dropped objects (count) 2 0.928 0.05 ,0.15
results Safety technical violations (count) 2.538 0.13 ,0.01
key unit performance metrics, we can also investigate the potential return for assigning Quality
additional personnel to the quality assurance function. We examine this idea through a
benefit-cost analysis.
assurance
The following assumptions were used in evaluating our results: staffing
.
The UMD-authorized quality assurance unit composition for each unit
represents the optimum quality assurance staffing requirement for that unit.
.
Although we understand that quality assurance personnel at each unit are culled 43
from the larger maintenance group staffing pool at each Air Force base, we do
not attempt to model the opportunity cost tradeoffs caused by this action.
. Once a person is assigned to an organization, there is a one-month lag between
their arrival at the duty station and them becoming a viable asset for that
organization.
.
All persons possessing the expertise in the reported staffing data are fully
capable, are assigned as assets under the maintenance group staffing structure,
and are not performing additional military duties outside of their expertise.
The quality assurance staffing variable can be interpreted as an elasticity value for
rate (non-count) dependent metrics. The elasticity formula is:
DY X %DY
· ¼
DX Y %DX
where Y, X are the variables being measured. For example, a one percent increase in
the quality assurance staffing value will yield a 0.7 percent decrease in the Break rate,
as shown in Table VI. Furthermore, when interpreting the impact on a count-type
metric, the marginal improvement is an amount as shown in Table VII – e.g. a 2 0.9
Dropped Object incremental change means that a 1 percent increase in quality
assurance staffing effectiveness will result in about one fewer dropped objects at each
base per month. (Note that Dropped Objects are maintenance-attributable items that
Variable Elasticity
Break 20.7
Cannibalization 22.4
Flying schedule effectiveness 20.4 Table VI.
Key task list pass 20.1 Percentage change in
Maintenance scheduling effectiveness 20.2 dependent variables for a
Quality verification inspection pass 20.1 one percent change in
Repeat 21.7 staffing effectiveness
Variable Incremental change Table VII.

Compiled incremental
Dropped objects 2 0.9 changes for COUNT
Safety/technical violations 2.5 metrics
JQME unintentionally separate from an aircraft while in flight, such as munitions, access
panels, or auxiliary fuel tanks.) A thumbnail Dropped Objects quality assurance
13,1 benefit-cost analysis for a typical aircraft maintenance organization provides some
guidance into the potential for Air Force cost avoidance. This example assumes an
annual personnel cost of $75,000 per added quality assurance noncommissioned
officer. We also assume that adding one noncommissioned officer to a typical quality
44 assurance unit results in a four percent increase in quality assurance staffing
effectiveness (applicable until the unit reaches 100 percent staffing effectiveness).
Furthermore, we assume the average Dropped-Object cost includes all costs across the
entire value chain: for replacement parts, the investigation, the resulting inspections
and corrective maintenance actions, etc. From Table VII, this four percent increase in
unit quality assurance staffing effectiveness translates to 3.6 fewer Dropped Objects
per base. Define the benefit b as the product of the cost per avoided Dropped Object,
the number of Dropped Objects avoided per month, the 16 bases, and the 12 months per
year, and note that the total annual cost of adding 16 noncommissioned officers is
$1,200,000.
We divide the Dropped Object benefit b by the cost of the additional 16 personnel to
derive the benefit-cost ratio:
$b Dropped Object Benefit
Benefit-Cost Ratio ¼
$1,200,000 noncommissioned officer cost
To simply recover the added personnel investment, Dropped Objects need only cost
$1,740 per incident. At a more realistic but still conservative cost of $5,000 per incident,
the benefit-cost ratio is over 2.8, equating to an annual cost avoidance of over $2.2
million. Consider that in 2003-2004, the 16 air bases we studied collectively averaged
about 1.5 Dropped Objects per month, with two high-activity bases experiencing about
2.7 per month (Moore, 2005). Even at 1.5 counts per month, the benefit-cost ratio is still
greater than 1.0 if Dropped Object costs average at least $4,200 per event. This example
alone suggests that increasing quality assurance unit staffing effectiveness (i.e.
assigning one more employee to each organization’s quality assurance unit) is justified
solely on the basis of decreasing Dropped Object counts.
4. Discussion
The magnitude and direction of the nine metric coefficients identified in Table V
generally agree with our intuition. Of note is the finding that Cannibalization rates
show a moderately strong negative relationship to unit quality assurance effectiveness
(cannibalization is the act of removing a serviceable part from one aircraft to replace an
unserviceable part on another). Cannibalization can be an effective supply strategy, but
it does create extra maintenance effort and thus should not be used to mask supply
system deficiencies. The regression result was supported by the correlation analysis
for individual units, which was most evident for one-to three-month lags (i.e. effective
quality assurance actions this month appear to mitigate cannibalization actions in the
next few months). On the other hand, Key Task List Pass rates, Maintenance Schedule
Effectiveness rates, and Quality Verification Inspection (QVI) Pass rates all showed
weak negative relationships to unit quality assurance effectiveness. These findings
were consistent with the mixed positive and negative individual unit correlation
analyses. This result could possibly be attributed to individual quality assurance unit
management strategies, as suggested by one Delphi panelist (Moore, 2005) – i.e. more Quality
effective quality assurance units are “tougher” when performing these critical assurance
inspections, and thus these rates would be expected at least initially to be lower. The
counter argument is that more effective quality assurance units will influence staffing
maintenance personnel to be more thorough in task accomplishment before calling the
inspectors, which should be reflected in a positive relationship between these metrics
and quality assurance staffing effectiveness. More research is needed. 45
Podsakoff and Organ (1986) mention potential problems with self-report based
research. For example, potential biases are inherent in perception-based surveys
because respondents can stage their responses in an overt or unintentional attempt to
affect the results. To mitigate this, we provided only basic instructional guidance on
our surveys. A second issue was the lengthy amount of time required to complete our
staffing effectiveness Delphi surveys. According to Podsakoff and Organ (1986),
respondents taking long surveys can experience “transient mood states” where a
consistent bias may be introduced across measures. To control for this, we provided a
“Save & Return Later” function in our Delphi surveys.
A final survey issue is the potential bias attributable to trait, source and methods.
For example, a Delphi panelist may have had consistently higher or lower expectations
on how effective somebody possessing their same specialty code may perform other
jobs (e.g. an electrician respondent may feel that an average electrician would be more
apt than people with other skills to expertly handle any type job they are assigned, and
thus may inflate their quality assurance effectiveness ratings for electricians. We
controlled for this potential bias by stressing that the panelists base their inputs on
“average” personnel performance, and by ensuring that our Delphi panel was diverse:
our panel members possessed different military ranks, different job specialty codes,
had different maintenance backgrounds, and represented different organizations.
From our research, we propose several recommendations. First, managers should
use the staffing effectiveness metric to understand the relationship between quality
assurance skill-task mismatching and organization performance. Air Force combat
aircraft maintenance units can use our derived matrix; other organizations should first
determine their own effectiveness matrix and associated unit performance measures.
Second, military aircraft maintenance quality assurance unit managers should perform
analyses to determine the strength and direction of any relationships between their
unit’s measured quality assurance staffing effectiveness and their unit’s performance
on the nine identified metrics. This will help them assess quality assurance staffing
effectiveness as a potential contributing factor for deficient areas indicated by their
metrics, and enable them to make the appropriate personnel decisions. Finally, we
believe this general method has applicability to any organization that seeks to
understand how their personnel resources correlate to their key performance
indicators.
Our future research efforts will concentrate on replicating this method and
performing benefit-cost ratio analyses with other military or civilian service
organizations with high-demand, low-density personnel resources. This will help
validate our method’s robustness and range of application. Secondly, the metric
relationships that were indicated in this study will be investigated through a structural
equations modeling technique to uncover potential additional linkages. We were
unable to obtain sufficient data in our initial research to pursue this approach.
JQME References
13,1 Dahlman, C., Kerchner, R. and Thaler, D. (2002), Setting Requirements for Maintenance
Manpower in the US Air Force, RAND Publishing, Santa Monica, CA.
“The Delphi Method.” (2005), available at: www.iit.edu/,it/delphi.html (accessed 11 September
2006).
Grier, P. (1998), “Board partly clears airman in F-15 crash”, Aerospace World, Vol. 81 No. 6, June.
46 Kutner, M., Nachtsheim, C., Neter, J. and Li, W. (2005), Applied Linear Statistical Models, 5th ed.,
McGraw-Hill Irwin, New York, NY.
Langewiesche, W. (1998), “The lessons of Valujet 592”, Atlantic Monthly, Vol. 281 No. 3, pp. 81-98.
McCormick, E., Pratt, D., Haunschild, K. and Hegdal, J. (1992), “Staffing up for a major program”,
Civil Engineering, Vol. 62 No. 1, pp. 60-2.
MacDonald, P., Francisco, M. and Azwar, A. (1996), “Internal quality assurance: lessons learned
from the PKMI Hospital”, Quality Assurance Methodology Refinement Series, June,
available at: www.qaproject.org/pubs/PDFs/INDO1.PDF

Moore, T. (2005), “Examining the impact of quality assurance manning practices in USAF
aircraft maintenance units”, MS thesis, The Air Force Institute of Technology,
Wright-Patterson AFB, OH (AFIT/GLM/ENS/05-18).
Oxley, L. (2000), “The ‘Top 10’ papers in econometrics”, invited paper presented to the
New Zealand Statistical Association Meeting, University of Canterbury, Christchurch,
September.
Podsakoff, P. and Organ, D. (1986), “Self reports in organizational research: problems and
prospects”, Journal of Management, Vol. 12 No. 4.
Srivastava, N. (2005), “The success story of quality assurance through mega packages at
TAPP-3&4”, An International Journal of Nuclear Power, Vol. 19 Nos 1-4, pp. 1-8.
Van den Beukel, A. and Molleman, E. (2002), “Too little, too much: downsides of
multifunctionality in team-based work”, Personnel Review, Vol. 31 No. 4, pp. 482-94.
Vessely, J. (1984), “Staffing for quality in nuclear power plants”, Transactions of the American
Nuclear Society, Vol. 46, pp. 689-90.
Zeng, D.D. and Zhao, J.L. (2005), “Effective role resolution in workflow management”, INFORMS
Journal on Computing, Vol. 17 No. 3, pp. 374-87.
Further reading
Air Force Instruction 21-101 (2004), Aerospace Equipment Maintenance Management, HQ
USAF/ILM. Pentagon, Washington DC. available at: www.e-publishing.af.mil/pubfiles/af/
21/afi21-101/afi21-101.pdf
Linstone, H. and Turoff, M. (1975), The Delphi Method: Techniques and Applications,
Addison-Wesley Publishing Co., Reading, MA.
Appendix Quality
assurance
Metrics Definition staffing
Abort Percentage of missions aborted in the air and on the
ground (an abort is a sortie that ends prematurely and
must be re-accomplished) 47
Break Percentage of aircraft that land “Code-3” (unable to
complete at least one of its assigned missions)
Cannibalization Number of cannabalization (CANN) actions (removal of
a serviceable part from an aircraft or engine to replace
an unserviceable part on another aircraft or engine
Combined mishap Aggregated count of all Class A, B and C mishaps both
flight and ground that are specifically related to
maintenance; also includes preventable aviation

maintenance-related injuries and incidents that did not
meet the $20K minimum reporting criteria
Dropped object and foreign object damage Two separate metrics that are aggregated counts of
occurrences of preventable Dropped Object counts and
Foreign Object Damage respectively (Dropped Object
is an item that falls off an aircraft (uncommanded)
while in-flight)
Deficiency reports submitted Measures number of instances technicians file material
deficiency reports on defective parts
Detected safety violations Counts of instances where QA observes personnel
commiting unsafe acts
Fix Shows how well the repair process is managed; is an
excellent tool for tracking “dead time” in aircraft repair
processes because it measures speed of repair and
equipment maintainability
Flying scheduling effectiveness Measures how well units plan and execute weekly
flying schedule
In-flight emergency Although not a primary metric and not tracked by all
ACC units (2 units in this study did not) it is
non-the-less an important one
Key task list pass Maintenance inspections QA performs on tasks that
are complex or that affect safety of flight (each time
maintenance accomplishes a KTL task, they must
notify QA to respond)
Mission capable Percentage of assigned aircraft mechanically capable
of meeting their assigned mission(s)
Maintenance scheduling effectiveness Measures unit’s ability to plan and complete
inspections and periodic maintenance on-time
according to maintenance plan
Maintenance/operations deviations Ratio of number of chargeable times an aircraft does
not meet its take-off window (within specific timing
standards) to number of sorties scheduled
Personnel evaluations pass Measures ability of personnel to perform tasks in their
duty position; a PE occurs when QA personnel perform
over-the shoulder evaluation of technician performing
a task for which the technician being inspected is
documented as being trained on
(continued) Table AI.
JQME Metrics Definition
13,1
Phase key task list pass Subset of overall KTL Pass Rate focused solely on
results of QA inspections performed on aircraft after all
maintenance is completed and before the aircraft rolls
out of a phase dock inspection
Quality verification inspection pass Inspection that QA personnel perform that can cover a
48 broad array of processes
Repeat When the same malfunction occurs on the very next
flight after it was repaired
Recur When the same malfunction for which an aircraft was
repaired, occurs on the 2nd-4th flights
Safety and technical violation Composite metric; number of times QA personnel
observe either: the person performing an unsafe act
(DSV); a person not following technical directives
(TDV); or an unsatisfactory condition (UCR)

Technical data violation Occurs when an individual performs a task, and either
doesn’t have technical data with him/her, or fails to
follow procedures according to technical data
Total non-mission capable for maintenance Average percentage of possessed aircraft that cannot
complete their primary assigned mission due to
maintenance reasons (except depot-type maintenance)
Technical order improvement submitted Number of instances when technicians submit TO
Table AI. improvement recommendations
Corresponding author
Alan W. Johnson can be contacted at: alan.johnson@afit.edu
To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
This article has been cited by:
1. D.E. Ighravwe, S.A. Oke. 2014. A non-zero integer non-linear programming model for maintenance
workforce sizing. International Journal of Production Economics 150, 204-214. [CrossRef]

Quality Assurance Staffing Impacts in Military Aircraft Maintenance Units OK

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Quality Assurance Staffing Impacts in Military Aircraft Maintenance Units OK

Uploaded by

Copyright:

Available Formats

Journal of Quality in Maintenance Engineering

Quality assurance staffing impacts in military aircraft maintenance units

References: this document contains references to 16 other documents.

Access to this document was granted through an Emerald subscription provided by

*Related content and download information correct at time of download.

METHODOLOGY AND THEORY Quality

Department of Operational Sciences, Air Force Institute of Technology,

Journal of Quality in Maintenance

organization—an expert assigned to the quality assurance unit is then unavailable to

2A6X4 Aircraft Fuel Systems Journeyman/Craftsman

assurance staffing effectiveness matrix to compute an overall monthly historical

Phase three: statistical analyses

14 2A651A 2A671 1 100

effectiveness scores for each of the 16 organizations, approximately 2000 total

Detected safety violations Positive Weak to moderate 4

Phase four: mitigating strategies

Dependent variable Coefficient Adj. R-Square p-value

Break (rate) 2 0.0073% 0.45 ,0.05

Variable Incremental change Table VII.

available at: www.qaproject.org/pubs/PDFs/INDO1.PDF

maintenance; also includes preventable aviation

(TDV); or an unsatisfactory condition (UCR)

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

You might also like