Performance Evaluation using

Control Charts
By Kapil Jaiswal

Abstract.............................................................................................................................................3
Objective of Study.............................................................................................................................4
Background .......................................................................................................................................4
Statistical Aspects .........................................................................................................................4
Basis of selection of control charts: ..........................................................................................6
Management Aspects ...................................................................................................................7
What is performance management and appraisal? .................................................................7
What are the common attributes evaluated in IT industry for performance appraisal? .........7
Methodology: ...............................................................................................................................8
Assumptions and Scope for this article ..................................................................................10
Control Charts for Points Achieved by developer.................................................................11
Control Chart Selection: ..........................................................................................................12
Limitation ................................................................................................................................13
Control Charts for Defects Injected by developer ................................................................15
Control Chart Selection: ..........................................................................................................16
Results.............................................................................................................................................19
Conclusions from Score based variable control chart: ........................................................19
Conclusion from Defect based attribute control charts:.....................................................19
Extending the results for different scenarios .................................................................................20
References ......................................................................................................................................21

Abstract
Organizations need to evaluate the performance of their employees at all levels to maintain a
competitive workforce and provide their employees growth opportunities.
Organizations further segregate the employees into different level of performance standards like
excellent, average or poor performers.
This article focus on use of control charts, a tool used for statistical process control, to create
these performance levels objectively, with the help of data gathered out of a project for a specific
duration.
This article also elaborates how the methodology can be extended for different scenarios and
organization requirements.

Objective of Study
The main objectives of this study are:

To objectively evaluate the performance of a group of employees, relatively against

each other using control charts.

To conclude statistically, if a group the employees can be segregated in the groups

like excellent, average and poor (or similar), based on their performances, using
control charts.

To review the reason of the employees lower than average and better than average
performance using control charts

Background
Statistical Aspects
[Mean: Simply stated, it is the average of all observations in a given sample. It is one of
the measures of central tendency. Measures of central tendency provide an idea about the
locations of the observations in the sample and the value about which they cluster.
Variance: The variance measures the scatter of the observations from the mean, the
larger the value, the greater the scatter. This is one of the measures of dispersion. In a given
sample, a variance provides an idea on the variability, or scatter of observations around a
given value, usually the mean.
Standard Deviation: The standard deviation also measures the variability of the
observations around the mean. It is the positive square root of variance. It has the same units
as the observations in the sample and thus is easier to interpret.] REF 2

Control Charts: [Control charts, also known as Shewhart charts (after Walter A.
Shewhart) or process-behavior charts, are among one of the tools for Statistical process
control used to make sure if a manufacturing or business process is in a state of statistical
control.] REF 4
[A control chart is a statistical (graphical) tool for monitoring the movement of a
continuing process. The values of quality characteristic are plotted along the vertical axis and
the horizontal axis represents the samples or subgroups from which this characteristic is
derived.] REF 2

[A control chart consists of:

Points representing a statistic (e.g., a mean, range, proportion) of measurements of a

quality characteristic in samples taken from the process at different times [the data]

The mean of this statistic using all the samples is calculated (e.g., the mean of the
means, mean of the ranges, mean of the proportions). A centre line is drawn at the
value of the mean of the statistic

Upper and lower control limits indicate the threshold at which the process output is
considered statistically 'unlikely' and are drawn typically at 3 standard deviation from
the centre line] REF4

Sample Control Chart:

12
10
8
Quality
Characterstic 6

Quality Charaterstic Value

Lower Control Limit (LCL)

Upper Control Limit (UCL)

Centre Line

0
1

Sample Values

Figure 1

[The values of the statistic are plotted on control chart, based on the assumption that the
sample has an approximately normal distribution.
The most common basis for analyzing whether a process is out of control is to check for any
sample statistic falling outside the control limits. The reason of such outliers is analyzed and
corrective actions are planned to bring back the process under control.
The causes of variation can be subdivided in to two groups common causes and special
causes.
Variability caused by special cause is something not inbuilt in the process. That is, it is not a
part of the process and does not affect all quality characteristic of all items.
Variability due to common causes is something inbuilt in the process. It is also referred to as
the accepted variation in a process.] REF 2

Basis of selection of control charts:

Figure 2

Management Aspects
What is performance management and appraisal?
[Performance management is the process through which the performance of the human
resources in an organization is identified, measured, managed and developed. Basically
organization tries to conclude how well employees perform and then to ultimately improve
that performance level.
Performance appraisal, on the other hand, is the continuous process of evaluating employees
performance. Performance appraisals are reviews of employee performance for a specified
duration. Appraisal is just one component of performance management.] REF [1]

What are the common attributes evaluated in IT industry for performance

appraisal?
In software development based IT project, the following attributes can be considered for
evaluating the performance of development team.
1) Number of defects injected
2) Schedule adherence
3) Process adherence
4) Total Efforts (may be in hours)
5) Any customized parameters like points/score given against a set of tasks, in a project
can also be used as a measurable parameter. I have used customized parameter in my
study.
Different roles, like testers, consultants, project managers etc. do have some different and
some common parameters, that are considered for their performance evaluation. Hence

this methodology can be extended to any role in IT or for that matter, to any other
industry.

Methodology:
Let us assume that we have a sample data of 10 employees along with a performance score
that they have got while working on a project. (Let us not go to in the details, how the process
of assigning score to the employees actually works, as of now).
The obvious solution to problem of ranking employees that comes to the mind is to just sort
the list on the basis of points in ascending or descending order and we will get the required
rankings.
But in reality, organizations do not rank everyone and evaluate them. Usually, employees are
segregated in few groups and get identified as excellent, average and poor performers. There
may be more categories for performance evaluation but the number of categories will always
be very small as compare to the total number of employees being evaluated.
Second obvious solution will be, to group employees based on their rankings/scores. We
assumed that we have got data of 10 employees, each having a score (points). Hence another
option is to (for example) divide this list into 3 groups of excellent (first 3 on the top), then
average (next 4 in the list) and poor (the bottom 3).
But this obvious choice might not be valid in all cases as the variability of the observations /
data may vary among different samples. There may be cases where the grouping will not be
that straight forward.
Control charts provide an appropriate solution to this situation.

If we plot the control chart with the available data of employees as mentioned above, it will
give us an idea about how scattered, the data is and if it is possible to form different groups of
employees based on their performance. The main benefit is that the outcome of this exercise
will not be based on a personal hunch but on the objectivity of data.
The groups based on performance levels on the control chart are identified as the area falling
between two straight lines parallel to X axis. One of these lines is generally drawn at mean of
sample and rest may be drawn at multiples of standard deviation from the mean.
Please refer to Figure 3 below. Digits 1, 2, 3, 4 and 5 depict the various regions/areas which
may be considered as different groups (levels) of performance.
Generally, organization utilizes this grouping based on performance (any derived score) for
deciding the yearly increment or bonus which is different across the groups (may be none for
lowest performance group) but same within the groups. The employees in different
performance groups are given different ratings/rankings but same in one group.

Figure 3

E.g. Employees in top group may be given a rating 1 (Excellent), the second group may get 2
(good), 3 (average), 4 (below average-poor) and the bottom group may be 5 (unacceptable).
Above image shows 5 different performance zones.
9

I have assumed that each developer be given a task of migration of web based reports from
one technology to another. These reports are of different size and complexity. A specified
score is assigned to each developer for conversion of each report, of a particular
size/complexity as per specified schedule.
E.g. For each small/easy report, 5 points will be given to the developer, 8 for
medium/average complexity and 10 for large/high complexity web page conversion, if they
complete this task in time.
Some developers will complete the assigned tasks in time, some cannot and some may
complete more than assigned.
In addition the points achieved, the defects injected (something that does not work as
expected) may also be monitored as an additional parameter to judge the developers
performance against quality.

Assumptions and Scope for this article:

3 levels of performance have been assumed for grouping the employees in this article
namely, above average level, average and below average performance. The control
limits have been drawn at 3 sigma and -3 sigma levels

Time sequence of fetching sample values is not considered and the sample values are
sequenced randomly.

The constant d2 in the formula for control limits is available till sample size 25. The
sample size of my data is 31 but I have use d2 for sample size 25.

It should also be noted that there may be other parameters which could be tracked but
I have considered only the above mentioned two parameters only (Points/Score and
Defects injected) for the scope of this article.

10

Control Charts for Points Achieved by developer

Developer

Points Achieved

Rahul

140

Nisha

65

75.00

Vinod

65

0.00

Mohit

120

55.00

Madhu

55

65.00

Raman

55

0.00

Shivraj

90

35.00

Puneet

130

40.00

Jyoti

130

0.00

Pooja

150

20.00

Rohit

85

65.00

Ismail

25

60.00

Lalit

100

75.00

Param

85

15.00

Sachin

220

135.00

Amitabh

100

120.00

Reetika

85

15.00

Yash

55

30.00

Kapil

65

10.00

Kunal

35

30.00

Vikram

200

165.00

Tej

70

130.00

Anurag

95

25.00

Pankaj

150

55.00

Manu

25

125.00

Abhishek

85

60.00

Ajay

85

0.00

Sanjay

65

20.00

Umesh

95

30.00

85

10.00

Shruti
Table 1- Sample Data for Points Scored

11

MR (Moving Range)

Relationship between sample data and control chart components (explained

earlier):

The quality characteristic for above sample is the individual points achieved and
plotted against the control limits.

The variability of the process is estimated from Moving Range (MR) found as the
positive difference of two successive observations.

The quality characteristic/statistic that is plotted against the control limits in a control
chart is Points achieved (first control chart in this article) and Defects Injected
(second control chart in this article) as mentioned above.

The control limits are set by finding the mean of sample and then by determining the
3 sigma values which are then plotted as upper (+3 sigma) and lower (-3 sigma)
limits.

Centre line is the mean of the sample taken.

UCL

X bar + 3*MR bar/ d2

130.246

LCL

X bar - 3*MR bar/ d2

52.657

MR bar

MR/ no. of MR values

50.833

Mean ( X bar)

Points Achieved/ No.

of Developers

91.452

Table 2: Formulas

Control Chart Selection:

We are inspecting each team member. Hence the sample size is 1.

The score is a variable type of data and hence calls for variable control chat.

The variable chart used with sample size one is X chart i.e. the values for each
developer (sample) is plotted against the mean and 3 standard deviations values.

12

Limitation
The time sequence aspect of sampling is not valid in this case. That means, the samples are
not drawn as in manufacturing industry, one by one. All the data for each developer was
maintained and taken together from the records. Hence Moving Range (MR) chart may not
be completely valid in this case and hence not created here.

13

14

50

100

150

200

250

Figure 4

Points Achieved

LCL

UCL

Developer Points

X bar

Control Charts for Defects Injected by developer

Developer

Tej
Amit
Deepti
Preet
Nidhi
Neha
Tushar
Aman
Naman
Mohit
Suresh
Sanjay
Ajay
Kapil
Puneet
Sunil
Nitin
Yash
Shruti
UCJaiswal
Vikas
Pankaj
Anurag
Saksham

15

Number of
Reports
(Sample
Size)

Defects

NC per Sample
Size
(Defects/Display
Converted)

UCL

0.33

1.02

0.20

0.89

0.20

0.89

0.33

1.02

2.00

1.86

0.25

0.95

0.00

0.45

0.67

1.26

0.17

0.85

0.20

0.89

0.25

0.95

0.25

0.95

0.67

1.26

0.25

0.95

0.40

1.08

0.50

1.15

1.50

1.67

0.14

0.82

0.67

1.26

0.67

1.26

0.67

1.26

0.50

1.15

1.00

1.45

0.25

0.95

LCL

0.13
0.0
0
0.0
0
0.13
0.97
0.05
0.4
5
0.37
0.0
4
0.0
0
0.05
0.05
0.37
0.05
0.19
0.26
0.78
0.0
7
0.37
0.37
0.37
0.26
0.55
-

Final LCL

Avg
NC=Total
Defects
(NC)/ Total
Web
Pages

0.45

0
0
0
0
0
0
0
0.04
0
0
0
0
0
0
0
0
0.07
0
0
0
0
0
0

0.05
Prachi
Parin
Jyoti
Pooja
*NC-Non Conformity

0.00

0.45

0.75

1.31

0.20

0.89

1.25

1.56

112

50

0.4
5
0.42
0.0
0
0.67

0
0
0
0

Table 3: Sample data of Defects

Relationship between sample data and control chart components:

The quality characteristic for above sample is the Defects Injected and plotted
against the control limits.

The control limits are set by finding the mean of sample and then by determining the
3 sigma values which are then plotted as upper (+3 sigma) and lower (-3 sigma)
limits.

Centre line is the average number of non conformities.

UCL

Avg NC + SQRT(Number of Defects/Sample

Size)

LCL

Avg NC - SQRT(Number of Defects/Sample

Size) or 0, whichever is higher

CL

Avg NC
Table 4-Formulas

Control Chart Selection:

It is to be noted that the occurrence of non conformities is assumed to follow a

Poisson distribution.

We are inspecting all reports created, for defects for each developer, which are
different in number, hence a variable sample size, which also varies the Upper and
lower control limits

16

This would be an attribute chart as we are monitoring the defects

U chart is most suitable in this case where we have got variable sample size to
monitor the non conformities per unit.

17

18

0.00

0.50

1.00

1.50

2.00

2.50

NC per Sample Size (Defects/Display Converted)

Figure 5

UCL

U Chart
Final LCL

Avg NC=Total Defects (NC)/ Total Web Pages

Results
The performance of developers has been objectively divided in to 3 performance groups
using control charts.
In both charts, some points (developers) are falling outside (or on) the control limits derived
based on sample data. The causes for the same were analyzed. In case of Points chart (X
chart), points falling below LCL, while in case of Defect chart (U Chart) points falling above
UCL, are of major concern (the below average performances in both cases).

Conclusions from Score based variable control chart:

Developers falling below LCL:

One of the reasons for fewer points is that some of the developers may have taken
more leave than other and hence achieved fewer points.

Another reason can be that some of them just joined the organization and hence
needed more time to adjust to company culture and technology.

Some can really be the low performers.

Developers lying above UCL:

The developers who work hard and are really good may come on the top. It should
also be analyzed if these developers follow any best practices which can be shared
with the team.

Conclusion from Defect based attribute control charts:

Developers lying above UCL:

19

The developers may be new and required further training to enhance his technical
skills.

Developer may not be using the standard checklists.

Extending the results for different scenarios

Can multiple parameters/goals be accommodated in the same chart?

o Yes. Assign different weights to different parameters and derive a weighted
score.
o X chart can be created as the weighted score will be a variable data.
o Calculate the mean, define the UCL/LCL and plot the chart.

Can UCL or LCL be set up by organization explicitly (MBO)? What if all developers
in the sample fall below organization standards?
o Yes. As described in OBJECTIVE section, organization can set UCL/LCL,
preferably (not mandatory) at any multiple of Standard deviation from mean
of data, e.g. 1 sigma, 2 sigma, 3 sigma etc and plot the graph. If the employees
in sample fall below the organization set specification limits, they may all be
grouped in to lower than average/poor performance level.

Can we create more than 3 performance groups?

o Yes. As described in OBJECTIVE section, each straight line drawn at a
multiple of standard deviation, parallel to X axis creates two performance
regions, one above and one below. Hence desired number of such lines should
be plotted to get as many performance groups, as are needed.

Can we apply this methodology to industry other than IT also?

o Yes. Only the parameters or attributes for evaluating the performance should
differ in that case.

20

References
1. http://www.sagepub.com/upm-data/45674_8.pdf
2. Fundamentals of Quality Control and Improvement (Second Edition) by AmitavaMitra
3. http://en.wikipedia.org/wiki/Control_chart

21