12 views

Uploaded by Laura Monica

article

- B.Sc. (H) Probability and Statistics 2011-2012
- One-Way Analysis of Variance by Abhishek Vijayvargiya
- Introduction to Statistical Process Control
- EXCEL Multiple Regression
- 01 - Exploratory Data Analysis
- traducir Analisis.docx
- Smith Nephew Pharma Equity Analysis
- 4th Sem. Asst. 2018-19
- Linear Regression
- Old Thesis
- Co Relation
- Linear Regression
- SM FINAL (2)
- Brad_Terry Model
- Final Report v.1-1
- Using Excel for Data Manipulation and Statistical Analysis
- tmp43D
- Co Inter
- partisi
- Transformer parallel

You are on page 1of 0

www.spectroscopyeurope.com

VOL. 18 NO. 2 (2006)

Introduction

It had been intended that this column

would be about PLS calibration, but

before we do this we need to fill a gap.

When we compute calibrations we

need some figure of merit to decide

if we have a good calibration. These

are calibration statistics which have not

been discussed in this column for a long

time.

r

2

?

Many people know and use a calibra-

tion statisticthe square of the product

moment correlation coefficient, r

2

so

why not just use this? There is a prob-

lem with r

2

but a demonstration may be

more persuasive than a lecture.

Figure 1 is a scatter plot of a calibration.

It is actually a simulated calibration using

data calculated from the model:

y = a + bx + e

with the es normally distributed and in

the data shown a = 0 and b = 1. The cali-

bration was produced by a calibration set

of 200 samples with a range of values

from 10 to 20 and a variability that gives

it an r

2

of 0.903.

Figure 2 shows a test of the calibra-

tion with a test set of 100 samples with

a range of values from 13 to 17. The r

2

was 0.625. Judging by the r

2

this is not a

good calibration. If we increase the range

of the test set to be between the values

of 12 and 18 the r

2

rises to 0.786, Figure

3. If we increase the range of the test set

to match the calibration (values of 10 to

20), Figure 4, then the r

2

increases to

0.911. The value of r

2

depends on the

range!

SEP

If r

2

is not the best choice for measur-

ing the merit of a calibration, then what

is? The answer is the standard error of

performance (or prediction), SEP, which

is a measure of the variability of the

difference between the predicted and

reference values for a set of validation

samples. The starting point is a set of

reference values r

i

and of predictions

p

i

for n samples in a validation set. Let

d

i

= r

i

p

i

be the difference between

reference and predicted for the ith

sample. Then the simplest way of calcu-

lating SEP is as the root mean square of

differences:

RMSEP = {(d

i

2

) / n}

where the sum, like all those that follow,

is over all samples, i.e. from 1 to n.

Because all the differences get squared,

it does not matter whether one uses

r

i

p

i

or p

i

r

i

in the calculation of d

i

.

This SEP is an estimate of what a typical

difference between prediction and refer-

ence values is likely to be when the cali-

bration is used for real. As such, it rightly

Back to basics: calibration

statistics

A.M.C. Davies

a

and Tom Fearn

b

a

Norwich Near Infrared Consultancy, 75 Intwood Road, Cringleford, Norwich NR4 6AA, UK;

b

Department of Statistical

Science, University College London, Gower Street, London, UK

Figure 1. Plot of the simulated calibration. r

2

= 0.90. Figure 2. Plot of results from a test set of 100 samples in the range

1317; r

2

= 0.625.

32 SPECTROSCOPYEUROPE

TONY DAVIES COLUMN

www.spectroscopyeurope.com

VOL. 18 NO. 2 (2006)

includes contributions from several

sources, including errors in the reference

measurements.

Unfortunately there are other versions

of the calculation, the most common

being the one which is corrected for bias,

SEP

b

.

First calculate the average difference

bias = (d

i

) / n

Then subtract this from each difference

in the calculation of SEP

b

, giving

SEP

b

= [{(d

i

bias)

2

} / (n 1)]

It can be shown that

*

the relationship

between the two is:

RMSEP

2

= SEP

2

b

+ bias

2

If we were able to know what the bias

would be when we predict real samples,

SEP

b

would be a useful statistic but if

the bias has changed since the calibra-

tion was done then there is no way of

knowing that it will not change again.

The tendency is for people to forget that

they have removed the bias! So either

use RMSEP or bias and SEP

b

. If the

bias is small, then RMSEP and SEP

b

will

be similar. The values of these statistics

for the previous examples are shown in

Table 1.

With a simple linear regression like this

one, the SEC, calculated from the differ-

ences between reference and predicted

values for the training data by a formula

like that for RMSEP but with a divisor

of n 2 to allow for the fact the predic-

tion line has been fitted to these data,

is a reliable predictor of what the SEP

should be on validation data. Here the

value of 0.621 is indeed close to the

observed SEPs. Unfortunately, in the

case of multivariate calibration with large

numbers of predictors, SEC is not a reli-

able guide to future performance, and

we need a validation set to estimate SEP

directly.

Conclusion

Chemometri c sof t ware wi l l proba-

bly continue to compute and display

r

2

values for ever and if it is there it is

impossible not to take note of it, but be

guided by the RMSEP or SEP

b

and bias

values. Calibrations should, of course, be

tested over the range in which they will

be used.

Figure 3. Plot of results from a test set of 100 samples in the range

1218; r

2

= 0.786.

Figure 4. Plot of results from a test set of 100 samples in the range

1020; r

2

= 0.911.

Variable Test set 1 Test set 2 Test set 3

Range 1317 1218 1020

r

2

0.625 0.786 0.911

RMSEP 0.615 0.615 0.616

SEP

b

0.616 0.616 0.618

bias 0.050 0.045 0.036

Table 1. Statistics for the examples shown in Figures 24.

*

For readers who enjoy some algebra!

Given that

RMSEP d n

i

i

n

=

=

2

1

/

or and nRMSEP d bias d n

i

i

n

i

i

n

2 2

1 1

= =

= =

( ) /

or nbias d

i

i

n

=

=1

and SEP d bias n

b i

i

n

=

= ( ( ) ) /( )

2

1

1

Square and multiply by (n 1)

( ) ( ) n SEP d bias

b i

i

n

=

=

1

2 2

1

Expand

( ) ( ) n SEP d bias d bias

b i i

i

n

= +

=

1 2

2 2

1

Sum individual terms

( ) n SEP d bias d bias

b i

i

n

i

i

n

i

n

= +

= = =

1 2

2 2

1

2

1 1

Substitute for

d nRMSEP d nbias

i i

2 2

= = and

i=1

n

note that

2

bias nbias

i

n

2

1 =

=

(n 1) SEP

2

b

= n RMSEP

2

2n bias

2

+ n bias

2

Thus, ignoring the difference between n

and n 1 and re-arranging

RMSEP SEP bias

b

2 2

= +

- B.Sc. (H) Probability and Statistics 2011-2012Uploaded byDharmendra Kumar
- One-Way Analysis of Variance by Abhishek VijayvargiyaUploaded byVipul Gupta
- Introduction to Statistical Process ControlUploaded byNilesh Jadhav
- EXCEL Multiple RegressionUploaded byAvra Ray
- 01 - Exploratory Data AnalysisUploaded byaladinsane
- traducir Analisis.docxUploaded byErick Franclin Fernández López Torres
- Smith Nephew Pharma Equity AnalysisUploaded bysazk07
- 4th Sem. Asst. 2018-19Uploaded byGOPAL SHARMA
- Linear RegressionUploaded bysvrbikkina
- Co RelationUploaded byAbdur Rajak
- Linear RegressionUploaded byJosue Perez
- Final Report v.1-1Uploaded byDamien Naidu
- Old ThesisUploaded bySharad Gajuryal
- SM FINAL (2)Uploaded byAakash Chharia
- Using Excel for Data Manipulation and Statistical AnalysisUploaded byRamon G Pacheco
- Co InterUploaded byaspendos69
- Brad_Terry ModelUploaded byTrung Hiếu
- tmp43DUploaded byFrontiers
- partisiUploaded byAshabul Kahfi
- Transformer parallelUploaded byGerald
- ANALISIS MELYUploaded byArensi Belo II
- Regression AssignmentUploaded bysarah
- statistikUploaded bySusana Moe
- out (2).pdfUploaded byJM Heramiz
- JMontelongo Takehome MidtermUploaded byjgbarajas2001
- LAB Human5Uploaded bymmmm
- Statistics-in-Research-Mama.docxUploaded byTwinkle Liboon
- Gb Paper Imi - 1 Sep 2013 ManishUploaded byManish Kumar
- Solution Ch.13Uploaded byBrooke Leverton
- _BIOSTATISTICS Uses in Some FieldsUploaded bykelvinkurt

- Impulse ResponseUploaded bydestiny710
- Resampling MethodsUploaded byghromis
- Lecture 11Uploaded byRaluca Muresan
- Bmgt230 Exam 3Uploaded byNathan Yeh
- ARIMA AR MA ARMA ModelsUploaded byOisín Ó Cionaoith
- Forecasting Using EviewsUploaded byLucia Spinei
- Output EviewsUploaded byPrasetyo Adi Priatno
- Ch4 Slides Part2fullUploaded byNgọc Huyền
- clogitUploaded bylsabetti
- 4503 Rc158 010d Machinelearning 1Uploaded byAhsanAnis
- AnovaUploaded byHimanshu Jain
- Panel_Data_1.docUploaded byAbdul Aziz
- Distributed Lag Model Br Bill NotesUploaded bySunil Kumar
- Learning Book 11 FebUploaded byTushar Agarwal
- STATA Training Session 2 Statistical Analysis in STATA.pdfUploaded byMatthew
- Two Way Mixed ANOVAUploaded bySunu Wibirama
- ECN-720, 603 Econometric Theory Spring 2014- OutlineUploaded bySumera Bokhari
- gretmman.pdfUploaded byfenomanana
- Stat.regression.worksheet November 14 SolutionsUploaded byanirban
- Sale of Tractors1Uploaded byFebin Koyan
- Problem_set_2_2016 Mixing Linear Algebra With Calculus and Probability TheoryUploaded bysilvio de paula
- Statistical FormulasUploaded bysatyendrashah
- STATA Basics Regression and Panal DataUploaded byFarhad
- Statistic Exam (MCQ)Uploaded byzephyryu
- Multivariate Analysis - MANOVAUploaded byPhenda Uka Ucinkk
- WHITE PAPER Multicollinearity in CSAT StudiesUploaded byanuj112358
- S-lab-2-2507-F11Uploaded byyana22
- ps12_sol.docUploaded byNataliAmiranashvili
- StatsUploaded byc684244
- 4-nn2-perceptronUploaded byKhalifa Bakkar