Standard Practice for

Recognizing Surrogate
Test Methods

AASHTO Designation: PP 112-211

First Published: 2021

Technical Subcommittee: Sc, Quality Assurance and Environmental

AAS HID
American Association of State Highway and Transportation Officials
555 12th Street NW, Suite 1000
Washington, DC 20004
Standard Practice for

Recognizing Surrogate Test Methods

AASHTO Designation: PP 1 12-2 11 AAS HID

First Published: 2021

Technical Subcommittee: Sc, Quality Assurance and Environmental

1. SCOPE

1.1. This practice describes a method to assess the capability of a surrogate test to be used as a
substitute for a standard test. The surrogate tests must demonstrate the capability to provide test
results similar to the standard test method with a minimum of additional variability. The test
results from the standard and the surrogate are compared and the error in their relationship is
quantified.

1.2. It may be desirable to use a surrogate test to monitor material quality during construction or
production, especially if the standard test method uses complex/expensive equipment, requires
time-consuming sample preparation and testing techniques, uses hazardous materials, or involves
destroying a portion of the material. This standard practice is intended to be applied for
construction quality control but may also be used for acceptance if specifically approved by the
Agency.

1.3. The procedure in this standard should be followed in the evaluation of potential surrogate test
methods and to set parameters for their use.

2. REFERENCED STAN DARDS

2.1. AASHTO Standards:

• R 9 , Acceptance Sampling Plans for Highway Construction
• R 10, Definition of Terms Related to Quality and Statistics as Used in Highway Construction
• R 61, E stablishing Requirements for E quipment Calibrations, Standardizations, and Checks

2.2. ASTM Standards:

• C670, Standard Practice for Preparing Precision and Bias Statements for Test Methods for
Construction Materials
• E2935, Standard Practice for Conducting E quivalence Tests for Comparing Testing Processes

2.3. NCSL International Standard:

• RP-1, E stablishment and Adjustment of Calibration Intervals

3. TERMINOLOGY

3.1. Definitions used herein are consistent with R IO; additional terms are provided as follows:

TS-5c pp 112-1 AASHTO

© 2021 by the American Association of State Highway and Transportation Officials.

All rights reserved. Duplication is a violation of applicable law.
3.1.1. standard test-A recognized standard test method that provides a test value and may include a
precision and bias statement.

3.1.2. surrogate test-A test that has been statistically correlated with the results from a standard test
method, but it is performed in a different manner than the standard test method.

4. PROCEDURE TO COMPARE STANDAR D AN D SURROGATE TESTS

4.1. The procedure involves three parts, 1) comparing the testing repeatability and reproducibility of
the standard test to the repeatability and reproducibility of the surrogate test, 2) identifying the
equivalence limit (E) for the test, and 3 ) measuring the equivalence (upper confidence limit
(E UCL) and lower confidence limit (ELCL)) in the means and variance relationship between the
standard and surrogate test. E very test method has inherent variability. In using a surrogate test
method, the variability of the surrogate test method adds an additional component of variability to
the standard test results. If that variability is minimal compared to the benefits of the surrogate
test, the surrogate test is acceptable for use. This procedure identifies and quantifies the added
variability, establishes a recommended limit to that variability and performs a hypothesis test to
test if the difference between the standard and surrogate test results exceeds the identified limit.

4.2. Step I-Compare the testing repeatability and reproducibility of the standard test to the
repeatability and reproducibility of the surrogate test. Step I provides confidence that the surrogate
test can be performed with similar variability by comparing the repeatability and reproducibility or
by comparing the standard and surrogate test methods using a statistical F-test on the variances.
R 9 Appendix X2 describes the F-test procedure, it is also a standard statistical method found in
standard computer spreadsheet and statistical analysis programs.

4.2.1. Identify single-operator and multilaboratory precision from the standard test method, if available.
See Section 4.2.2 if the standard test method does not have a precision statement. Compare the
maximum ratio of the variance of the standard and surrogate procedures to an Fcritical of 1.84
(Fcritical from an F-test, assuming alpha= 0.05 (a 95 percent confidence limit) and numerator and
denominator values of n= 30).
Note 1-It is preferred to use this practice with a standard test method that has an established
precision statement.

4.2.1.1. Identify single-operator and multi-laboratory precision of the surrogate test using ASTM C670.

4.2.1.2. If the repeatability standard deviation ( S ) and reproducibility standard deviation (SR) from the
r

standard test is greater than the surrogate tests, report the repeatability standard deviation (Sr) and
reproducibility standard deviation (SR) from the surrogate test. Perform an F-test to compare the
variances as noted in Section 4.2.1., but note that the surrogate test has less variance.

4.2.1.3. If the repeatability standard deviation (S,.) and reproducibility standard deviation (SR) from the
standard test is less than the repeatability standard deviation (Sr) and reproducibility standard
deviation (SR) from the surrogate tests, perform an F-test to statistically compare repeatability and
reproducibility variances. Confirm that the reproducibility and repeatability variance between the
results obtained from the standard test and the surrogate test are statistically the same at 95 percent
confidence, assuming the numerator and denominator values of n= 30.

4.2.2. lf the standard test method does not have a precision and bias statement compare the variability of
the tests by performing a statistical F-test on a minimum of 30 split samples that have been tested
using the surrogate and standard tests. Confirm that the variance, based on the standard deviation
(s2) between the results obtained from the standard test and the surrogate test, are statistically the
same at 95 percent confidence.

TS-5c pp 112-2 AASHTO

© 2021 by the American Association of State Highway and Transportation Officials.

All rights reserved. Duplication is a violation of applicable law.
4.2.3. Report which test method (standard or surrogate) has higher variance (Section 4.2.1. Sr2 and Si or
Section 4.2.2 split sample s2), and the results from the F-test (F and p-value) from the surrogate
and standard tests.

4.3. Step 2-Identify E and n. The appropriate equivalence limit, E, shall be no more than two times
the reproducibility standard deviation of the standard test (2 * SR), unless another value is
specifically approved by the agency implementing the surrogate test. E should also be identified
considering the practical significance of the test values. Identify the minimum number of split
samples (n) using ASTM E2935 Section X2.l , with sigma= S, or split samples, alpha= 0.05, and
E, for a power level of at least 90 percent.

Note 2-See Section Xl .1.3 for an example of determining n using Power curves.

4.4. Step 3-Perform a hypothesis test to identify if the error in the relationship (means and slope)
between the standard and surrogate test are within the defined equivalence limit (£) defined in
Section 4.3 using a minimum ( n) number of samples as identified in Section 4.3 . Step 3 defines
the relationship between the standard and surrogate test results and provides a limit to the measure
of potential error between the two test results. The procedure is similar to simple regression in that
it evaluates the linear statistical relationship between the standard and surrogate test method.
When performing this test the residuals should be checked for normality.

4.4.1. Perform a means equivalence two one-sided test (TOSI) using split samples in accordance with
Section 7 of ASTM E2935 and report the equivalence upper confidence limit (EUCL) and
equivalence lower confidence limit (ELCL) for the difference in means. Using the same data
perform a paired t-test and an F-test as described in R 9 and report the p-values.

4.4.2. Perform a slope equivalence test using split samples in accordance with ASTM E2935 Section 8
and report the equivalence upper confidence limit (E UCL) and equivalence lower confidence limit
(ELCL) for the slope.

4.4.3. Plot the dot plot for residuals, residuals versus x, and line slope comparison as noted in ASTM
E2935 Section 8.3 to evaluate any outliers or heteroscedasticity.

4.5. Report the results from Steps 1, 2, and 3, including graphs.

5. PROCEDURE TO IDENTIFY VALIDATION FREQUENCY

5.1. A calibration and standardization frequency for the surrogate test shall be defined following the
requirements in R 61 and RP-1. Consider the frequency of calibration required for the standard test
method and any uncertainty growth in the surrogate test method. It is recommended to start with a
very high frequency (> 1 per 30 samples) and adjust the frequency if the results remain consistent.

5.1.1. The surrogate test should describe the range of values and types of materials that were used in
calibrating the test. The surrogate test should not be used outside the range of values used in
calibrating the surrogate test.

5.1.2. Recalibration should be performed if the material being tested varies greatly from the material that
was used to calibrate the surrogate test. If there is a concern of material changes, a quick one-test
check can be performed to confirm the relationship is the same. The process in Section 4 should be
repeated if the materials change outside the range previously considered or there is concern the
relationship is not remaining consistent over time.

TS-5c pp 112-3 AASHTO

© 2021 by the American Association of State Highway and Transportation Officials.

All rights reserved. Duplication is a violation of applicable law.
6. SURROGATE ACCURACY REPORTING

6.1. Precision-Records of precision shall be maintained and made available on request. Records
should include total number of samples, sample type/description, and origin. Results provided
should include the mean and range of data values, and the results of the three-step analysis
performed in Section 4.

6.2. Bias-Bias should be addressed by any algorithm that was used to develop the surrogate test
value, so it should not be a consideration, unless the standard test method bias must be considered
in the referee testing. In that case, issues related to bias should be documented and made available
on request.

6.3. Limitations-Record of any limitations or conditions that increase variability of the test shall be
maintained and provided with the equipment.

7. KEYWORDS

7.1. Regression; statistical comparisons; surrogate test.

APPENDIX

(Nonmandatory Information)

X1. EXAMPLES OF SURROGATE COMPARISON

X1.1. E xample 1: iCCL (incremental creep for cracking at low temperature) surrogate test that provides
continuous low-temperature PG similar to BBR using a DSR.

X1.1.1. For this example only, continuous low-temperature PG (LTPGc) of R 29 is considered. The
surrogate test for determining LTPGc is iCCL (incremental creep for cracking at low temperature).

X1.1.2. Step I-Compare the testing repeatability and reproducibility of the standard test to the
repeatability and reproducibility of the surrogate test.

X1.1.2.1. Table X.l presents the repeatability and reproducibility coefficient of variation for the standard
and surrogate test. The testing variability for the surrogate test are less than the standard test. A
comparison of the variance between the standard and surrogate provides a maximum F value of
1.452/1.402 1.07 for S,. and 3 .42/1.9 2 3 .2, as compared to Fcritical of 1.84. Therefore, the
= =

variances are statistically similar for repeatability and the variances are statistically different for
reproducibility. This is for information only since the variance of the surrogate is less.

TS-5c pp 112-4 AASHTO

© 2021 by the American Association of State Highway and Transportation Officials.

All rights reserved. Duplication is a violation of applicable law.
Table X1 .1-Comparison of Repeatability and Reproducibility

Single-Operator Precision Multilaboratory Precision

Acceptable Coefficient Acceptable Coefficient

Coefficient of ofVariation of2 Test Coefficient of ofVariation of2 Test
Variation,% Results,% Variation,% Results,%

Property (ls limit) (d2s limit) (ls limit) (d2s limit)

Continuous Low­ 1 .45" (I .40 )h 4.1 0" ( 4.0) " 3.40° (1 .9 )h 9.62" (5.3 )h
Temperature PG
Notes:

The first set of values are the precision estimates for continuous low-temperature PG, calculated from the precision estimates of m-value provided in T 313,
according to ASTM D4460.

The second set of values(in parentheses) are the single-operator and multilaboratory coefficient of variation of iCCL from the 2019 round robin study.

X1.1.3. Step 2-Identify E andn.

X1.1.3.1. Eis limited to two times the SR. Eis identified using the d2s of the BBR SR: -22 *9 .621100= 2.11
degrees. Then divide 2.11 by 2 to get the ±E value. For simplicity, Eis determined to be
±1 degree. Since PG grades are typically identified in ranges of 6 degrees (i.e. -22, -28, -3 4),
±1 degree also appears practical.

X1.1.3.2. The minimum number of split samples,n, is determined by developing power curves for the
condition of sigma (S,)= 0.6, alpha= 0.05, and E= 1 as shown in Table XI.2 and Figure Xl. I.

Table X1 .2-Table for Computing Minimum Sample Size,n

Standard Deviation 0.6 z Alpha

Consumer Risk 0.05
Design (#Samples)
Equivalence Limit
Sample Size, n 10 20 30 50
Std Error 0.1 90 0.1 34 0.1 1 0 0.085
True Diff Power Power Power Power
0 0.9997 1 .0000 1 .0000 1 .0000
0.2 0.9949 1 .0000 l.0000 l.0000
0.4 0.9354 0.9977 0.9999 1 .0000
0.6 0.6784 0.9776 0.9776 0.9989
0.8 0.2773 0.5718 0.571 8 0.7618
0.0500 0.0500 0.0500 0.0500
1 .2 0.0035 0.0003 0.0003 0.0000

X1.1.3.3. The minimum sample size is the value that provides a minimum90 percent power at the value of
sigma. In Figure Xl.2, the absolute difference of 0.6 (x-axis) provides a power of90 percent
(y-axis) near a value ofn= 20. Therefore, minimumn= 20. Note that a value ofn greater than 20
is also acceptable.

TS-5c pp 112-5 AASHTO

© 2021 by the American Association of State Highway and Transportation Officials.

All rights reserved. Duplication is a violation of applicable law.
 1.0

0.9

0.8

0.7

0.6

Oi
� 0.5
Cl.

0.4

50
0.3
n =

0.2

0.1

0.0
0.0 0.2 0.4 0.6 0.8 1.0 1.2

True Absolute Difference

Figure X1 .1-Power Curves to Identify Minimum Sample Size, n

X1.1.4. Step 3-Perform a hypothesis test to identify if the error in the relationship (means and slope)
between the standard and surrogate test are within the defined equivalence limit (£) using at least
the minimum number of split samples (n) determined in Section Xl .1.3 .

X1.1.4.1. Table X1.3 shows a portion of the 60 samples from the data, and the results of the TOST analysis
comparing the means. Table XI .3 also shows thep-value for a standard F-test and paired t-test for
the same data. The results of the TOST analysis are within the ELCL and EUCL. The results of
the F-test and paired t-test also indicate that the variance and means are statistically similar.

TS-5c pp 112-6 AASHTO

© 2021 by the American Association of State Highway and Transportation Officials.

All rights reserved. Duplication is a violation of applicable law.
Table X1.3-TOST ExamEle
BBR iCCL
Sample# PG Grade True PG Surrogate Di ff
58H-34 -36.472563 -36.728571 -0.256008
2 46-34 -36.043323 -36.6 -0.5566769
3 58H-34 -35.395729 -36.013333 -0.6176047
4 58H-34 -35.38461 5 -35.91 571 4 -0.5310989
5 58-34 -33.93617 -32.981667 0.95450355
6 64-28 -31 .093971 -30.034667 1 .05930434
7 64-28 -31 .065604 -30.893333 0.1 722708
8 58-28 -30.965174 -30.2 0.76517388
9 58H-28 -30.957746 -30.954 0.00374648
10 64-28 -30.943396 -30.408571 0.5348248

55 76-22 -23.787234 -23.56181 8 0.22541586

56 64-22 -23.730769 -22.91 0.82076923
57 64-22 -23.701493 -23.74 -0.0385075
58 76-22 -23.538462 -24.56 -1.0215385
59 64-22 -23.358491 -22.8 0.55849057
60 64-22 -22.735888 -22.392 0.3438877

Average -27.772803 -27.687044 0.0857598

Std Dev, Sd 3.53791446 3.698681 85 0.7133766

n 60 60
Degrees ofFreedom (f l , f2)) 59 59

D (Means Difference) 0.0857598

SD (Standard Error of the Means Difference) 0.09209652
a (consumer risk) 0.05
E (Equivalence Limit) loC
F- test
t (one tailed) -1 .2960657 p = 0.734
ELCL -0.0336033 >-1 Paired t-Test
EUCL 0.20512294 < 1 p = 0.356

X1.1.4.2. Table X1.4 shows a portion of the 60 samples from the data, and the results of the analysis
comparing the slope. The results of the slope analysis are within the ELCL and EUCL.

TS-5c pp 112-7 AASHTO

© 2021 by the American Association of State Highway and Transportation Officials.

All rights reserved. Duplication is a violation of applicable law.
Table X1 .4- Slope Equivalence Test Example

pred pred pred

BBR iCCL Xi-Xbar Yi-Ybar Sxx Syy Sxy y Resids. y y

Sample Ave LT (Xi- (Yi- (xi-Xbar) LCL UCL

# True PG PG Xbar)"'2 Ybar)"'2 *(yi-Ybar) slope slope
1 -36.47 -36.73 -8. 70 -9 . 04 75.69 81. 75 78.66 -36.79 0. 04 -36.41 -37.19

2 -36.04 -36.60 -8.27 -8. 91 68.40 79. 44 73. 71 -36.34 -0.18 -35. 9 8 -36.72

-35.40 -36.01 -7.62 -8.33 58.11 69 . 33 63.47 -35.66 -0.24 -35.33 -36.01

4 -35.38 -35.92 -7. 61 -8.23 57. 9 4 67. 71 62.64 -35.65 -0.18 -35.32 -36.00

5 -33. 9 4 -32. 9 8 -6.16 -5.29 37. 99 32.63 -34.14 0. 80 -33.87 -34.42

6 -31.09 -30.03 -3.32 -2.35 11.03 5. 51 7. 80 -31.16 0. 78 -31.02 -31.31

-31.07 -30.89 -3.29 -3.21 10.84 10.28 10.56 -31.13 0. 17 -30. 9 9 -31.28

8 -30. 9 7 -30.20 -3.19 -2.51 10.19 6.31 8.02 -31.03 0. 57 -30.89 -31.17

9 -30. 96 -30. 9 5 -3.18 -3.27 10.14 10.67 10.41 -31.02 0. 05 -30.88 -31.17

10 -30. 9 4 -30.41 -3. 17 -2.72 10.05 7. 41 8. 63 -31.00 0. 41 -30.87 -31.15

55 -23.79 -23.56 3. 9 9 4.13 15.88 17.02 16. 44 -23.52 -0.03 -23.69

56 -22. 9 1 4.04 4. 78 16.34 22.82 19 . 31 -23. 46 0. 38 -23.27

57 -23.70 -23. 74 4.07 3. 9 5 16.58 15.58 16.07 -0.22 -23.61 -23.24

58 -23.54 -24.56 4.23 3.13 17. 93 9 . 78 13.24 -23.26 -0. 90 -23. 44 -23.06

59 -23.36 -22.80 4.41 4.89 19 . 49 23.88 21.57 -23.07 0. 19 -23.26 -22.87

60 -22.74 -22.39 5.04 5. 30 25.37 28.04 26.67 -22.42 0.02 -22.64 -22.19

Ave. -27.77 -27.69 -l.36E-l5 3. 73E-15 738.49 807.13 757. 8 l. 87E-13

Var. 12.517 13.680

R R2

0. 982 0. 963 25 alpha 0. 05

1.046 1.372

El 0. 9 slope

theta E2 I.I slope

0.808 1.672

phi Part I Part 2

0.021 0.439 0.09 7

LCL
(teta) UCL(teta)

0.786 0. 829

ELCL EUCL
(beta) (beta)

1.002 1.092
ELCL> EUCL<
0. 9 l. l

LCL(a) UCL(a)

Note: Values have been rounded to fit in the table.

X1.1.4.3. Figure Xl .2 shows the graphs of the surrogate vs standard tests, dot plot of residuals and the
slopes of the standard/surrogate relationship. The residuals were checked for normality both
statistically ( Shapiro-Wilkp-value = 0.2576) and with a Q-Q plot as shown in Figure Xl .3 .

TS-5c pp 112-8 AASHTO

© 2021 by the American Association of State Highway and Transportation Officials.

All rights reserved. Duplication is a violation of applicable law.
 l-
-40 - 35 - 30 -2 5 - 0
•
••

•
·�·
•
•
• • -2 5

..
•

_J
u -3 0
!-:::! "r..
.....:.1

-3 5
•
••

-4 0
BBR

(a) Continuous Low-Temperature PG

-1.50 -1.00 -0.50 0.00 0.50 1.00 1.50

(b) DOT Plot for Residuals

1
- 35 -30 -25 0

-25

-3 0

-3 5

UCL Slope -- Prediction Slope -- LCL Slope

(c) Line Slope Comparison

Figure X1 .2-Graph Plots for Slope Equivalence Test

TS-5c pp 112-9 AASHTO

© 2021 by the American Association of State Highway and Transportation Officials.

All rights reserved. Duplication is a violation of applicable law.
 I Distributions
[Residuals

0.5

-0.5

-1

0.02 0.05 0.1 0.2 0.3 0.5 0.7 0.8 0.9 0.95 0.98

Normal Quantile Plot

[9uantiles
100.0% maximum 0.98
99.5% 0.98
97.5% 0.9695
90.0% 0.798
75.0% quartile 0.29
50.0% median -0.03
25.0% quartile -0.2975
10.0% -0.65
2.5% -0.94425
0.5% -0.96
0.0% minimum -0.96

�ummary Statistics J
Mean 0.0005
Std Dev 0.4908844
Std Err Mean 0.0633729
Upper 95% Mean 0.1273089
Lower 95% Mean -0.126309
N 60

Figure X1 .3-Residual Q-Q Plot Example

TS-5c pp 112-10 AASHTO

© 2021 by the American Association of State Highway and Transportation Officials.

All rights reserved. Duplication is a violation of applicable law.
X1.1.4.4. Example 1 shows a surrogate test method that is within the defined ELCL and EUCL limits.

X1.2. Example 2: Theoretical Surrogate Test

X1.2.1. Figure Xl .4 shows a low correlation between the standard and surrogate tests. Based on the
regression slope, this test would not be an acceptable surrogate under this standard.

0.9 •

0.8 •

0.7
QJ
+-' y = 0.1986x + 0.2826
ro 0.6 •
b.O R2 = 0.0497
0 0.5 •
:I...
:I...
0.4
::J
(/) 0.3 •
•
0.2 • •
•
0.1 • •

0
0 0.2 0.4 0.6 0.8 1 1.2

Standard

Figure X1 .4-Theoretical Surrogate Test

1 This provisional standard was first published in 2021.

TS-5c pp 112-11 AASHTO

© 2021 by the American Association of State Highway and Transportation Officials.

All rights reserved. Duplication is a violation of applicable law.