2010 5th International Conference on Industrial and Information Systems, ICIIS 2010, Jul 29 - Aug 01, 2010, India

Least Square based Parabolic Regression to

determine Calibration Constant in Measurement
P. Mallick, C. Nayak, T. B. Behera
Integrated Test Range
Defence R & D Organisation
Chandipur, Balasore-756025, India
E-mail: p_mallick123@rediffmail.com

Abstract— This paper highlights the use of Calibration Constant
of Test & Measuring (T&M) equipment derived during
calibration process, in measurement inference. The equipment
user is much interested to find the true value of measurand
against the measured quantity. In this paper, the Best-fit-curve is
proposed to model a set of test results obtained during
calibration by Least Square Error (LSE) based parabolic
regression. Calibration Constant becomes more dominant rather
than the detailed multi page test results to determine the actual
value of measurement with certain degree of accuracy. This
reduces interpolation error while calculating the true value other
than testing points and also the measurement time. The Test set-
up is automated and the measured parameters are traceable to
national standards.
Keywords- Best-fit-curve, Calibration Constant, Interpolation,
Least square error, Parabolic function, Regression
I. INTRODUCTION
Calibration is the process by which integrity of the quality
of measurement remains intact. Before taking any final
recommendations or decisions, the due importance is to be
given to calibrated measurement data & the associated
uncertainty. Due to the properties of material and so frequent
use of T&M equipment, the performance may get deteriorated
with respect to time. There are also other factors which
predominates the measurement result in a measurement
process such as gain, offset and linearity errors. Several
methods have been adopted for linearization of equipment
performance to get better measurement inference. Internal
processing circuit of equipment is being modified and updated
to facilitate the same. Electrical noise may introduce certain
degree of non-linearity in the low level precision signal
measurements process. Noise level shall be minimized at the
source by proper designing source coding of software filters
and moreover by the use of appropriate shielding & grounding
techniques. Equipment manufacturer provides the discrete test
points for calibration of various performance parameters.
During recalibration, the calibration engineer brings out the
standard results of equipment under test & a certificate of the
test result to that effect is used to user for its application.
The practical application of the calibration data in real time
measurement plays a crucial role to assure the quality in
measurement process. The engineer handling the T&M
equipment takes decision depending upon calibration results.
But it becomes cumbersome for the engineer to go through
calibration certificate in detail and find correction and
uncertainties involved for certain measurement points other
than all test points. Therefore, modeling of T&M equipment
performance by mathematical equation and its curve reduces
the difficulties of user in real time measurement. The simplest
and the most common type of best fit curve is a straight line.
As discussed in [4], data interpolation is achieved by the best
fit straight line to the data points obtained during calibration,
which is represented by straight line equation as
1
Where, ‘y’ represents the standard or actual value and ‘x’ is
the measured value. ‘ a ’ and ‘ b ’ are the calibration constants
determined by least square method. The method of least
squares assures that the best fit is one which the sum of squares
of the residual is minimum [6],[9]. The calibration line as fitted
is then used to obtain the actual value ‘ y ’ for measured value
of ‘ x ’.
However, linear assumption for interpolating points may
not appropriate for accurate measurement. In [15], the
cumulative distribution function in the transferred domain of a
sum of log normal random variables is approximated using
least squares method via piecewise linear functions. This
method increases the complexity in the computation with more
number of basic functions.
Probability density function (PDF) based progressive
polynomial calibration [1] evaluates calibration co-efficient
directly for smart sensor linearization is one of the possible
solution for linearization.
The least square based parabolic regression method
presented in this paper seems to be an attractive solution for
computation of calibration constants in measurement because it
requires minimal time with less computational complexity for
accurate measurement inference.
II. LEAST SQUARE BASED PARABOLIC
REGRESSION & CALIBRATION CURVE
Polynomial regression is used to fit data points by the
equation
2
Co-efficients ‘ a ’, ‘ b ’, ‘ c ’…. are determined to make the
curve best fit for data points. When ‘ a ’ & ‘ b ’ are to be fit
[10], the equation describes a line, and polynomial regression
is identical to linear regression as explained by Eq. (1).

1
978-1-4244-6653-5/10/$26.00 ©2010 IEEE 334
 2010 5th International Conference on Industrial and Information Systems, ICIIS 2010, Jul 29 - Aug 01, 2010, India

The T&M equipment readings (i.e. measured values) Eq. (12) can be written as,
against a set of standard values are taken along ‘X’-axis and
standard values are along ‘Y’-axis. A set of data series are 13
represented by the parabolic equation as Where,
3
The calibration curve obtained by fitting above equation is
then used to obtain the actual value of ‘ y ’ for measured value
of ‘ x ’.
, 14
The method of least squares assures that the best fit is one,
which the sum of squares of the vertical residual is a minimum
[9]. Here ‘ a ’, ‘ b ’& ‘ c ’ are considered as calibration
constants for a particular set of measurement. Eq. (3) may be
written as
; , , 4
For a set of paired data , ; 1,2, … … … , , the
residual error 'is defined by , 14

; , , 5

Here the residual error is considered as the vertical offsets.

As per the least square theorem, Sum of Squared Errors (SSE)
‘ ’ is to be minimum. By partial differentiation with respect to
‘ a ’, ‘ b ’& ‘ c ’ and equating these partially derivatives to
zero, we obtain,
14
2 0 6

2 0 7 Linear matrix equation AB=C can be solved by Gaussian

elimination method. This method reduces the computation
complexity in comparison to other method, however rounding
2 0 8 errors get accumulated. As rounding error does not impact
much in a second order polynomial regression, which is the
reason of using this method in solving above matrix for fitting
The above equations lead to the following normal calibration curve.
equations,
Mean and variance of curve fitting error are defined by,
9 ∑
15
∑
16
10 1
For any measurement other than the calibrated points, the
regression equation as explained in Eq. (3) is used to compute
11 actual value of measurement. The correlation co-efficient ‘r’ is
computed [16] to study the behaviour of regression curve on
fitting of sampled paired data as obtained during calibration
Or, in matrix form, process.
∑ ; , ,
17
∑
Here, ‘ ’ is the mean reading of standard value of a set of
12 paired data , .
The correlation coefficient and variance of curve fitting
error indicates the closeness of curve to the fitted data. Amount

2
335
 2010 5th International Conference on Industrial and Information Systems, ICIIS 2010, Jul 29 - Aug 01, 2010, India

of data not considered for the curve fitting is being estimated
by,
∑ ; , ,
1 18
∑
The accuracy of calibration curve as fitted is closely
associated with the value of . To minimize the effect of
random measurement errors, multiple point calibration is
preferred although it takes more time.
III. EXPERIMENTAL SET UP
The test is undertaken with TRMS (True Root Mean
Square) Multi meter “RishMulti 18S”. UUT (Unit Under Test)
performance is compared with its higher standard, calibrator
Fluke 5520A. The cable loss is taken care by using the standard
IV. TEST RESULT AND DISCUSSION
The test result of UUT at different test points and
associated estimated value (both linear and parabolic
regression using least square method) is represented in Table
I. Using calibration data obtained as average value on UUT
and the corresponding value on standard, the best fit curve is
plotted based on least square parabolic regression (Fig. 2).
From Table I, it is found that zero estimated error is more
in linear regression in comparison to parabolic regression,
whereas error at 6V & 9V is more in parabolic with respect to
linear estimation. “Fig. 3” shows the trend of curve fitting
error.
TABLE I. CALIBRATION DATA AND ESTIMATED VALUE

measurement cable. The measurement parameters are traceable Average Value on Measured yest yest
to National Standard. To get better measurement performance Value Standard Error (Parabolic) (Linear)
of UUT, Test Uncertainty Ratio (TUR) is maintained at 4:1 at on UUT (V) (V) (V) (V) (V)
each calibration test points. To minimise the human error, the xi yi xi - yi
0 0 0 0.0730 -0.3844
test procedure is partially automated through calibration
5.25 5.0 0.25 5.0158 4.9736
software MET/CAL version 7.2[11]. Five sets of 6.15 6.0 0.15 5.8840 5.8921
measurements are taken at the respective test points and the 7.28 7.0 0.28 6.9828 7.0454
parameter uncertainty is estimated as per Guide to the 8.34 8.0 0.34 8.0223 8.1272
Expression of Uncertainty in Measurement (GUM). The GUM 9.19 9.0 0.19 8.8620 8.9947
approach considers a standard deviation and the confidence 10.21 10.0 0.21 9.8768 10.0357
interval in which the measurand is supposed to lie with a given 11.34 11.0 0.34 11.0102 11.1890
level of confidence [4]. The distribution of measurement 12.48 12.0 0.48 12.1634 12.3524
results is assumed to be Gaussian. 13.46 13.0 0.46 13.1626 13.3526
15.23 15.0 0.23 14.9856 15.1590
The selection of test points is being guided by the 17.24 17.0 0.24 17.0845 17.2104
manufacturer. Using manufacturer’s specification, the 19.12 19.0 0.12 19.0753 19.1291
calibration curve is fitted as explained above. Although there is 19.8 20.0 -0.20 19.8019 19.8231
a scope of selecting PASS/FAIL depending upon the allowable
TABLE II. INTERPOLATION RESULT
limit with logic conditions, it is always preferred to limit the
test result to 70% of the allowable limit to extract better data Average Value on yest(int) yest(int) yint - yintP yint - yintL
for best fit calibration curve. As calibration simulates the Value on Standard (Parabolic) (Linear)
performance of UUT by a mathematical model, the UUT (V) (V) (V) (V) (V) (V)
xint yint yintP yintL
environment factor is also considered by controlling the stable 2.496 2.25 2.3970 2.1629 -0.147 0.0871
environmental condition of calibration facility. The data flow 3.695 3.50 3.5301 3.3866 -0.0301 0.1134
in a calibration system is shown in “Fig. 1”. Serial and Parallel 5.494 5.35 5.2506 5.2226 0.0994 0.1274
port interface unit of Standard communicates with central 7.494 7.25 7.1920 7.2638 0.058 -0.0138
processor in both direction for executing the calibration 10.494 10.35 10.1607 10.3256 0.1893 0.0244
software and sends the stimulus for UUT and measured data 12.493 12.05 12.1766 12.3256 -0.1266 -0.2756
are stored in the database of central processor. Statistical 14.456 14.0 14.1855 14.3691 -0.1855 -0.3691
16.28 16.0 16.0782 16.2306 -0.0782 -0.2306
computation is performed by MATLAB software.
17.67 17.5 17.5375 17.6493 -0.0375 -0.1493

20
STANDARD Actual Reading
18
Estimated Reading
16
I/O Keypad Processor and Central
Data Base

and Display Command Unit Processor 14

for
Standard Value(V)

12
Calibration
Serial & Parallel Software
10

Interface Unit Execution 8

I/P for process initiation 4

UUT 2

Figure 1. Data flow for Calibration System 0

0 2 4 6 8 10 12 14 16 18 20
Measured Value in UUT(V)

Figure 2: Calibration Curve for Best fit line

3
336
 2010 5th International Conference on Industrial and Information Systems, ICIIS 2010, Jul 29 - Aug 01, 2010, India

1.2
On computation, the values for ‘ a ’, ‘ b ’& ‘ c ’were
computed as 0.0730, 0.9217 and 0.0038 respectively. From the
Curve Fitting Error
1 tabulated data, standard deviation of error was found 0.1753.
Considering the interpolation measurement, the standard
0.8
deviation of error for parabolic regression was found 0.1236.
Curve Fitting Error(V)

0.6 Similarly, same set of data points are used for computing
the best fit line using straight line equation Y = a + b X. The
values for ‘ a ’ & ‘ b ’ obtained are -0.3844 and 1.0206
0.4

0.2 respectively. Standard deviation of interpolation measurement

error for linear regression was found to be 0.1847. Standard
0 uncertainty of linear fit error is 50% more in comparison to
parabolic fit error. On comparing both the Linear and Parabolic
-0.2
0 2 4 6 8 10 12 14 16 18 20 Regression calibration data, it is quite evident that parabolic
Measured Value in UUT(V)
regression gives better result in comparison to linear curve. In
Figure 3: Curve Fitting Error Trend assessing goodness of fit [10], the runs test is a simple and
robust method for determining whether data differ
It is observed that the best fit curve error is much more at symmetrically from a theoretical curve. From Graph 4, it is
test point 19V among all test points. The interpolation result at found that interpolation residuals from estimated value follow a
arbitrary measurement points is tabulated in Table II. The systematic pattern, with negative residuals for the first and last
comparison result between two regressions indicates the few points and positive residuals for the middle points.
behaviour of curve trend.
The “r2” value (square of the correlation coefficient) for
“Fig. 4 and 5” both indicate the interpolation curve fitting parabolic regression is found 0.9639. So, 3.6% of tabulated
behaviour of T&M equipment in a particular measurement data is not considered for best fit curve plotting. However, it is
range. observed that the calibration curve considering three calibration
constants (‘ a ’, ‘ b ’& ‘ c ’) gives more accurate result in
18
interpolation.
Actual Reading
16
Estimated Reading V. CONCLUSION
14
The statistical approach as proposed was found one of the
Standard Interpolation Value(V)

12 solution to get more accurate calibration constants for a

particular range of measurement. Least square based parabolic
10 regression to determine calibration constants gives better
8
inference in comparison to linear regression. However,
obtained result is restricted with the following conditions:
6
a) Calibration procedure is followed as per
4 manufacturer’s specification.
2
b) Statistical analysis is beng carried out based on least
2 4 6 8 10 12
Measured Interpolation Value(V)
14 16 18
square approach considering vertical residuals only.
The calibration engineer derives calibration constants “a”,
Figure 4: Interpolation Curve “b” and “c” with detail certificate. Performance of T &M
equipment is now modeled as the best fit parabolic equation
0.2 Y = a + bX + cX2. For any measurement value of “X”, the user
0.15
Curve Fitting Error can estimate the actual value using the equipment model
equation. This reduces the operator’s computational error for
0.1 interpolation measurement inference and eliminates its
uncertainty component during the measurement process. The
Curve Fitting Error(V)

0.05
same concept can be extended to get fixed calibration constant
0
for particular equipment considering the total range into
different continuous segments, which will be of future concern
-0.05 for analysis.
-0.1
ACKNOWLEDGMENT
-0.15 The authors are greatly thankful to Sri S.P. Dash, Scientist-
H(Outstanding Scientist), Director, ITR for allowing us to use
-0.2
2 4 6 8 10 12 14 16 18 the Calibration Facility of the establishment to achieve the
result and also thankful to Dr. S.N. Banerjee, Additional
Interpolation Value(V)

Figure 5: Interpolation Curve Fitting Error Trend Director & Sri B.K. Paltasingh, Scientist-F, GD (QA) for
providing constant guidance and moral support.

4
337
 2010 5th International Conference on Industrial and Information Systems, ICIIS 2010, Jul 29 - Aug 01, 2010, India

REFERENCES [8] ‘International Vocabulary of Basic and General Terms in Metrology’

[1] Pereira J. M., Postolache Octavian, and Silva Girao P.M.B., “PDF-
Based Progressive Polynomial calibration Method for Smart Sensor
Linearization,” IEEE Transactions on Instr. & Meas. , Vol. 58, No. 9,
September 2009, pp. 3245–3252.
[2] Stone D., Ellis J., “Calibration and Linear regression Analysis: A Self-
Guided Tutorial,” CHM314 Instrumental Analysis, Department of
Chemistry, university of Toronto
[3] Mallick P., “A Case Study on Decision Making with Expression of
Uncertainty in Measurement,” in Proc. IEEE Region 10 Colloquium
and 3rd International Conference on Industrial and Information Systems,
India, December 2008.
[4] Mallick P., Nayak C., and Behera T.B., “Statistical Approach for
Calibration Constant in Measurement,” in Proc. NSI-34 and
International Conference on Instrumentation, India, January 2010.
[5] Schoukens Johan, Nemeth Jozsef G., Vendersteen Gerd, Pintelon Rik,
and Crama philippe, “Linearization of Nonlinear Dynamic Systems,”
IEEE Transactions on Instr. & Meas. , Vol. 53, No. 4, August 2004, pp.
1245–1248.
[6] Antona G.D., “The Full Least-Squares Method,” IEEE Transactions on
Instr. & Meas. , Vol. 52, No. 1, February 2003, pp. 189–196.
[7] ‘Guide to the Expression of Uncertainty in Measurement’ ISO
Document, 1993.
ISO Document, 1993 (Geneva: International Organisation for
Standardisation)
[9] Taylor.J.K, “Statistical Techniques for Data Analysis,” CRC Press.
[10] Harvey J. Motulsky, and Lennart A. Ransnas, “Fitting curves to data
using nonlinear regression: a Practical and nonmathemetical review,”
University of California, pp. 365-374.
[11] “Fluke Metrology Sofftware User’s Manual,” Fluke Corporation, 1999.
[12] ‘The Expression of Uncertainty and Confidence in Measurement
(M3003)’, Edition 2, January 2007.
[13] Kimothi. S.K, “The uncertainty of Measurement-Physical and
Chemical, Metrology impact and Analusis,”ASQ Quality Press.
[14] “Expression of the Uncertainty of Measurement in Calibration,” EA-
4/02, European Co-operation for Accreditation, pp. 53-58, December
1999.
[15] Zhao L., and Ding J., “Least Squares Approximation to Lognormal Sum
Distribution via Piecewise Linear Functions,” ICIEA 2009, pp. 1324-
1329.
[16] Spiegel Murray R., “Theory and Problems of Probability and Statistics,”
Schaum’s Outline Series, Metric Editions, 1980.

5
338