International Research Journal of Advanced Engineering and Science

ISSN (Online): 2455-9024

Statistical Tests for Pairwise Comparisons of Signal-

to-Noise Ratios: The Nominal the Best Case
A. Bizimana1, J. D. Domínguez2, H. Vaquera3
1
University of Rwanda, College of Science and Technology –Kigali, Rwanda E-mail: bizimanal2003@yahoo.fr
2
Centro de Investigación en Matemáticas –Unidad Aguascalientes, México E-mail: jorge@cimat.mx
3
Colegio de Postgraduados –Campus Montecillo, México E-mail: hvaquera@colpos.mx

Abstract— We propose statistical tests for pairwise comparisons of the asymptotic distribution of the estimate of the signal-to-
signal-to-noise ratios when the response variable is “the nominal the noise ratio. Statistical tests for pairwise comparisons of signal-
best” case. A Monte Carlo study and an illustrative example on real to-noise ratios are presented. A Monte Carlo study and an
data are provided. illustrative example on real data are provided.
Keywords— Asymptotic distribution, multivariate delta theorem, II. SIGNAL-TO-NOISE RATIO FOR THE NOMINAL THE BEST
pairwise comparisons, signal-to-noise ratio, statistical test.. CASE
I. INTRODUCTION Let y1 , y2 ,…, yn be a realization of iid random variables
Robust parameter design is one of the most creative and Y1 , Y2 ,…, Yn normally distributed with mean µ and
effective tools in quality engineering. This tool works by
identifying factor settings to reduce the variation in products variance σ . In many cases, it is of interest to achieve a target
2

or processes. Robust parameter design had been practised in value for the response, say y = T , while the variation is
Japan for many years before it was introduced to the United minimum [7]. Deviations in either direction are undesirable. In
States of America by its originator Genichi Taguchi in the this case, Taguchi recommends the following signal-to-noise
mid-1980’s [1]. ratio:
One of the central ideas in the Taguchi approach to  µ2 
parameter design is the use of the performance criterion that SNRT = 10 log 10  2  . (1)
σ 
he called Signal-to-noise ratio (SNR) for variation reduction
Its estimate is obtained as
and parameter optimization. The signal-to-noise ratio is a
performance measure that combines the mean response and  y2 
variance [2]. The extend to which maximization of such
SNR T = 10 log10  2  ,
s 
(2)
 
criterion can be linked with minimization of quadratic loss
1 n
was considered in [3]. where y = ∑ yi is the sample mean and
The signal-to-noise ratio that is used depends on the goal n i =1

( )
of the experiment. Different goals of the designed experiment 1 n 2

are as follows: s2 = ∑
n − 1 i =1
yi − y is the sample variance.
1. The nominal the best: The experimenter wishes for the
Note that (2) can be written as
( ) − 10 log
response to attain a specific target value.
( s ).
2
2. The smaller the better: The experimenter is interested in SNR T = 10 log10 y 10
2
Kacker [8] pointed out
minimizing the response.
that in cases where the response variance and mean are
3. The larger the better: The experimenter is interested in
independent, one or more factors (tuning or adjustment
maximizing the response.
factors) can be used in order to eliminate the response bias,
The signal-to-noise ratio has generated many controversies
as seen by the discussions on Box’s paper [4] and the panel that is, the adjustments result in E ( y ) = T . If one assumes an
E ( y − T ) reduces
2
discussions edited by Nair [5]. Different studies have proposed additional model, the loss function
statistical improvements to the signal-to-noise ratio, for
example [6]. to Var ( y ) . As a result, the estimate of the signal-to-noise ratio
Multiple comparisons of treatments is one of the most reduces to SNR T = −10 log10 ( s 2 ) . If the mean y is set at a
important topics in designed experiments. In the literature, the
concept of multiple comparisons of treatments based on target value, then maximizing SNRT is equivalent to
signal-to-noise ratios is not studied. The objective of this minimizing log10 ( s 2 ) [2]. When the variation in the
paper is to propose statistical tests based on signal-to-noise
ratios for pairwise comparisons of treatments when the log10 ( s 2 ) component is larger than the variation in the

( ) component, SNR
response variable is the nominal the best case. We initially 2
define the signal-to-noise ratio for the nominal the best case. log10 y T is dominated by the variation in
In addition, for performing statistical inference, we determine

148

A. Bizimana, J. D. Domínguez, and H. Vaquera, “Statistical tests for pairwise comparisons of signal-to-noise ratios: The nominal the best
case,” International Research Journal of Advanced Engineering and Science, Volume 2, Issue 1, pp. 148-152, 2017.
 International Research Journal of Advanced Engineering and Science
ISSN (Online): 2455-9024

log10 ( s 2 ) . Therefore an analysis of the signal-to-noise ratio  2 

 µ 
essentially reduces to an analysis of log10 ( s 2 ) [1]. ∇ f (θ ) =  . (7)
 1 
− 2 
III. ASYMPTOTIC DISTRIBUTION OF THE STIMATE OF THE  σ 
SIGNAL-TO-NOISE RATIO Applying the multivariate Delta theorem leads to
In order to conduct the tests of hypothesis for pairwise   y2   µ 2  a
n  ln  2  − ln  2   ~
comparisons of signal-to-noise ratios, it is important to know   s 
  σ  
the distribution of the estimate of the signal-to-noise ratio. The
multivariate delta theorem [9] is applied for determining the   2 T  σ 2  2 
    0  
asymptotic distribution of the estimate of the signal-to-noise µ   n  µ ,
N  0,  (8 )
ratio.   1   2σ 4   1  
Result 1. Asymptotic distribution of SNRT   − 2   0 − 
  σ   n   σ 2  
Let y1 , y 2 , … , y n be realizations of iid random variables
i.e.,
Y1 , Y2 ,… , Yn normally distributed with mean µ and variance
 y  2
 µ 2  a  4σ 2 2 
σ 2 . Then the estimate of the signal-to-noise ratio for the n  ln  2  − ln  2   ~ N  0, + , (9 )
  s   σ   nµ 2 n 
nominal the best case, SNRT , is asymptotically distributed as  
or equivalently,
 10   µ 
2
normal with mean µSNRT =   ln  2 
and variance  y2  a   µ 2  4σ 2 2 
 ln10   σ  ln  2  ~ N  ln  2  , 2 2 + 2  . (10 )
s 
   σ  n µ n 
 10   4σ
2
2  2
σ SNR
2
=   2 2 + 2  [10].
T
 ln10   n µ n  It follows that
Proof  10    µ  4σ 2 2 
a 2
SNR T ~   N  ln  2  , 2 2 + 2  , (11)
 ln10    σ  n µ n 
The estimate of the signal-to-noise ratio for the nominal
the best case, say SNRT , can be written as a
 y 2   10   y 2  where ∼ stands for asymptotically.
SNR T = 10 log10  2  =  ln  .
 s   ln10   s 2 
( 3) Therefore, the estimate of the signal-to-noise ratio is
    asymptotically distributed as normal, this is,
( )
a
SNR T ~ N µ SNRT , σ SNR
2
, (12 )
Let θ = ( µ , σ 2
) be a vector of unknown parameters of the T

normal population such that the vector θɵ = y , s 2 ( ) is its where

 10   µ 
2
estimator. We recall that the variance-covariance matrix of
µ SNR =    2  and
ln
θɵ is given by ([9])
T
 ln10   σ 
σ 2   10   4σ
2 2
2 
 0  σ SNR
2
=   2 2 + 2 • (13)
( )
Var θɵ = 

n .
2σ 4 
(4)
T
 ln10   n µ n 
 0 
 n 
Let f : R 2 → R be a bivariate function such that IV. STATISTICAL TESTS FOR PAIRWISE COMPARISONS OF
SIGNAL-TO-NOISE RATIOS
 µ2 
f (θ ) = f ( µ , σ 2 ) = ln  2  . (5) In this section, exploiting the properties of the asymptotic
σ  normality and the Central Limit Theorem ([11], [12]), we
present statistical tests for pairwise comparisons of signal-to-
The corresponding partial derivatives respect to µ and σ 2 are, noise ratios when the response variable is of the nominal the
best case. We begin by considering two independent normal
respectively,
populations with mean µ i and variance σ i2 , i = 1, 2.
∂f (θ ) ∂  µ2  2 ∂ f (θ ) ∂  µ2  1
= ln  2 = an d = ln  2 =− 2. 6) Suppose that y1 and y 2 are two independent samples of
∂µ ∂µ  σ  µ ∂σ 2
∂σ 2
σ  σ
The gradient vector is sizes n1 and n 2 , respectively, drawn from the above
mentioned populations such that:
Sample 1: y1 = y11 , y12 ,… , y1n1 and

149

A. Bizimana, J. D. Domínguez, and H. Vaquera, “Statistical tests for pairwise comparisons of signal-to-noise ratios: The nominal the best
case,” International Research Journal of Advanced Engineering and Science, Volume 2, Issue 1, pp. 148-152, 2017.
 International Research Journal of Advanced Engineering and Science
ISSN (Online): 2455-9024

Sample 2: y2 = y21 , y22 ,… , y2 n2 . The statistical test in case µ1 , µ2 , σ 1 and σ 2 are known is given
Let SNRT1 and SNRT2 represent the corresponding signal-to- by
noise ratios. The corresponding estimates of signal-to-noise
z=
( SNR T1 ) (
− SNR T2 − SNRT1 − SNRT2 )
, (18 )
ratios are SNRT1 and SNRT2 respectively. It is desired to test σ SNR T1 − SNR T2

the hypothesis
H 0 : SNRT1 = SNRT2 against H 1 : SNRT1 ≠ SNRT2 , (14 ) and the statistical test when µ1 , µ2 , σ 1 and σ 2 are unknown is
or equivalently, ( SNR T1 ) (
− SNR T2 − SNRT1 − SNRT2 )
(19 )
H 0 : SNRT1 − SNRT2 = 0 against H 1 : SNRT1 − SNRT2 ≠ 0 . (1 5 ) t= .
σ SNR T1 − SNR T2

Result 2. Mean and standard deviation of SNRT1 − SNRT2 Under H 0 , SNRT1 − SNRT2 = 0 , and the statistics in (18) and
Let y1 = y11 , y12 ,… , y1n and y 2 = y 21 , y 22 ,… , y 2 n
1
be 2
two (19) reduce to the following expressions.
independent samples of sizes n1 and n2 , respectively, drawn The statistical test in case µ1 , µ2 , σ 1 and σ 2 are known is given
from two independent normal populations with mean µ i and by
variance σ i2 , i = 1, 2. Under H0, the mean and standard SNR T1 − SNR T2
z=
deviation of SNRT − SNRT are asymptotically zero and σ SNRT − SNRT
1 2 1 2

 10  4σ 1 4σ 22  10   y1   10   y 2 
2 2 2
2 2
 2 2 + 2 + 2 2 + 2, respectively [10].

 ln10  n1 µ1 n1 n2 µ 2 n2   ln 2 −   ln 2
 ln10   s1   ln10   s2 
=
 10  4σ 1 4σ 22
2
Proof 2 2
  + + + 2
 ln10  n1 µ1 n1 n2 µ 2 n2
2 2 2 2 2
In fact,
 10   µ1   10   µ 2 
2 2
µ SNR = µ SNRT − µ SNRT =  ln  2  −  ln  2   y2   y2 
  ln  12  − ln  22 
T1 − SNRT2 1 2
 ln 10   σ 1   ln 10   σ 2   s1  s 
   2 
= SNRT1 − SNRT2 = 0. (16 ) = . ( 20 )
4σ 12 2 4σ 22 2
+ + +
n12 µ12 n12 n22 µ 22 n22
The standard deviation of the difference of SNRT and 1
The statistical test in case µ1 , µ2 , σ 1 and σ 2 are unknown is
SNRT2 , say σ SNR T1 − SNRT2
, is determined as follows:
given by
σ SNR = σ SNR + σ SNR  10   y1   10   y 2 
2 2 2 2
T1 − SNRT2 T 1 T 2
  ln −   ln
 ln10   s1   ln10   s2 
2 2
SNR T − SNR T
 10   4σ   10   4σ 
2 2

t= =
2 2
2 2 1 2
=    + +   +
1 2
 σ SNR − SNR  10  4 s1
2
4 s22
 ln 10   n µ 2 2 2
n1   ln 10   n µ
2 2 2
n2  T1 T2 2
+ 2 + +
2
1 1 2 2
  2
 ln10  n 2 y n1 n 2 y 2 n22
 10  4σ 1 4σ 2
2 2

(17 )
2 2 1 1 2 2
=   + + + •
 ln 10  n1 µ1
2 2
n1
2
n2 µ 2
2 2 2
n2 y  2
y  2

ln  21  − ln  22 
 s1   s2 
   
Result 3. Statistical tests for comparing SNRT and SNRT 1 2 =
2 2
• ( 21)
The statistical test for comparing SNRT and SNRT in the case 4 s1 2 4 s2 2
1 2 + 2 + + 2
2 2 n 2 2 n
 y2   y2  n y
1 1 1 n y 2 2 2
ln  21  − ln  22 
 s1   s2 
µ1 , µ 2 , σ 1 and σ 2 are known is     , and a a

4σ 12 2 4σ 22 2 Under H 0 , z ∼ N ( 0, 1) and t ∼ tν , where

+ + +
n12 µ12 n12 n22 µ22 n22 ν = n1 + n2 − 2 represents the degrees of freedom of the t
y  2
y  2 distribution. The null hypothesis, H 0 , is rejected if z > zα or
ln  21  − ln  22  2
 s1   s2 
the statistical test becomes     when α
4s12
2 4s 2 2
t > tα , where z α is the quantile of the standard normal
,ν 2
2
+ 2 + 22 + 2 2 2
2 n1 n 2 y n2
n y
1 1 2 2 α
µ1 , µ 2 , σ 1 and σ 2 are unknown [10]. distribution and tα is the quantile of the t distribution
,ν 2
2

Proof with ν degrees of freedom.

150

A. Bizimana, J. D. Domínguez, and H. Vaquera, “Statistical tests for pairwise comparisons of signal-to-noise ratios: The nominal the best
case,” International Research Journal of Advanced Engineering and Science, Volume 2, Issue 1, pp. 148-152, 2017.
 International Research Journal of Advanced Engineering and Science
ISSN (Online): 2455-9024

V. MONTE CARLO STUDY OF THE PROPERTIES OF THE
PROPOSED TESTS
Monte Carlo simulations are performed to evaluate the
performance of the proposed statistical tests in terms of test
sizes and powers. Sample means and sample variances are
used to determine the estimates of signal-to-noise ratios.
Simulation under H 0 , this is, simulation with equal population
parameters ( µ X = µY and σ X = σ Y ) permits estimating the
test size. Under H1 , simulations are conducted after applying
an increment ∆ to the population parameters. Simulations with
different values of population parameters give the estimates of
power tests.
A. Procedure for Monte Carlo simulation
The simulation process has been conducted according to the
following procedure:
1. From two independent normal populations, X and Y ,
such that X ∼ N ( µ X , σ X2 ) and Y ∼ N ( µY , σ Y2 ) , simulate
two independent samples of sizes n X = nY = 10.
2. Calculate the sample means and sample variances;
X , Y , s X2 and sY2 .
3. Calculate the estimates of the signal-to-noise ratios; SNR X
and SNR Y .
4. Based on asymptotic normality of the estimates of the
signal-to-noise ratios, simulate MC = 10000 replicates of
∆ = 0.001, 0.01, 0.1, 1. The increment ∆ = 0 implies equal
parameters.
B. Results
Table I shows the estimated sizes of the test statistic. The
population parameters used are µ X = µY = 35 and σ X = σ Y = 2.
The row entries represent the proportion of times H 0 was
rejected at α = 0.05 under H 0 , this is, the proportion of
times H 0 is wrongly rejected. The test size is very close to the
significance level. Moreover, it seems that the sample size
does not affect the value of the test size.
TABLE I. Estimated Type I error rates of t test for various sample sizes.
Sample size Type I error
10 0.0499
20 0.0500
30 0.0498
60 0.0496
Table II contains the estimated powers obtained in
changing the population means and population variances
simultaneously. In this case, the population parameters used in
simulations are: µY = µ X + ∆ µ and σ Y = σ X + ∆σ . The row
entries represent the proportion of times H 0 is rejected at
α = 0.05 under H1 , this is, the proportion of times H 0 is
correctly rejected.

( ) TABLE II. Estimated powers of t test for various sample sizes and various
a
SNR X ∼ N µ SNR X , σ µ2 and increments, changing the population means and population variances
SNR X
simultaneously.
N (µ ). Four configurations of sample
a
SNRY ∼ SNRY
, σ µ2 Sample ∆ µ = 0.001 ∆ µ = 0.01 ∆ µ = 0.1 ∆µ = 1
SNRY

sizes are used: n = 10, 20, 30, 60. size ∆σ = 0.001 ∆σ = 0.01 ∆σ = 0.1 ∆σ = 1
10 0.0503 0.0799 0.9990 1
5. For each replicate, conduct a t test for the null hypothesis 20 0.0589 0.8642 1 1
H 0 : SNR X − SNRY = 0, and count the number of rejections 30 0.1365 1 1 1
(# Rejections). 60 0.9981 1 1 1

# Rejections
6. Determine the rejection rate: . Table III contains the estimated powers, obtained in
MC changing the population means and maintaining population
The parameters used in Step 1 are determined by applying variances at constant values. In this
an increment ∆ according to the following scheme: case, µY = µ X + ∆ µ and σ Y = σ X . The row entries represent the
1. Simultaneous change of population means and population
variances. The population parameters are determined as proportion of times H 0 is rejected at α = 0.05 under H 1 .
follows:
TABLE III. Estimated powers of t test for various sample sizes and various
µY = µ X + ∆ µ and σ Y = σ X + ∆σ ; where ∆ µ and ∆σ are
increments, obtained in changing the population means and maintaining the
increments in population mean and population variance, population variances at constant values.
respectively. Sample size ∆ µ = 0.001 ∆ µ = 0.01 ∆ µ = 0.1 ∆µ = 1
2. Changing the population means and maintaining the 10 0.0499 0.0504 0.0622 1
population variances at constant values. In this scheme, the 20 0.0497 0.0549 0.4549 1
population parameters are determined as follows: 30 0.0503 0.0800 0.9996 1
60 0.0596 0.08449 1 1
µY = µ X + ∆ µ and σ Y = σ X .
3. Changing the population variances and maintaining the Table IV contains the estimated powers, obtained in
population means at constant values. In this case, the changing the population variances and maintaining population
population parameters are determined as follows: means at a constant value. In this
µY = µ X and σ Y = σ X + ∆σ . case, µY = µ X and σ Y = σ X + ∆σ . The row entries represent the
Four configurations of increments are used: proportion of times H 0 was rejected at α = 0.05 under H 1 .

151

A. Bizimana, J. D. Domínguez, and H. Vaquera, “Statistical tests for pairwise comparisons of signal-to-noise ratios: The nominal the best
case,” International Research Journal of Advanced Engineering and Science, Volume 2, Issue 1, pp. 148-152, 2017.
 International Research Journal of Advanced Engineering and Science
ISSN (Online): 2455-9024

signal-to-noise ratio is determined. We propose statistical tests

TABLE IV. Estimated powers of t test for various sample sizes and various for pairwise comparisons of treatments with regard to the
increments, obtained in changing the population variances and maintaining the
population means at constant values.
signal-to-noise ratio when the response variable is the nominal
∆σ = 0.001 ∆σ = 0.01 ∆σ = 0.1 ∆σ = 1
the best case. The correction to these pairwise comparisons
Sample size
can be done using the Bonferroni inequality as stated by
10 0.0499 0.0841 0.9997 1
20 0.0602 0.9000 1 1
Chang [15]. The correction consists in applying the adjusted
30 0.1471 1 1 1 level of significance and adjusted p − value.
60 0.9995 1 1 1 Illustrations of the proposed tests based on simulation and
on real data are presented. The values of the estimated test
Results in tables II, III and IV show that the estimated sizes are displayed in Table I. Tables II, III, and IV display the
powers of t test increase as the increments increase. Effects of values of the estimated test powers according to the three
sample sizes to the estimated powers of t test are remarkable. scenarios presented in the paragraph on Procedure for Monte
For the same value of increment in the population parameters, Carlo simulation. The results of the Monte Carlo simulations
the proposed test detects a significance difference between show that the statistical tests we propose preserve the test size
two values of signal-to-noise ratios, with high power, if the when simulations are conducted under H 0 and have excellent
corresponding sample size is also high. powers when simulations are conducted under H1 .
VI. REAL EXAMPLE
REFERENCES
We consider the problem of a robust design conducted on [1] J. Bérubé and C. F. J. Wu. “Signal-to-noise ratio and related measures in
a chemical process [11]. The target value is set at T = 6. This parameter design optimization: An overview,” Sankhya: The Indian
the best value obtained for the proportion of impurities in [14]. Journal of Statistics, Series B, vol. 62, issue 3, pp. 417-432, 2000.
The data obtained for the first two runs of the experiment are [2] R. H. Myers, A. I. Khuri and G. Vining. “Response surface alternatives
to the Taguchi robust parameter design approach,” The American
in table V. Statistician, vol. 46, issue 2, pp. 131-139, 1992.
[3] R. V. León, A. C. Shoemaker and R. N. Kacker. “Performance measure
TABLE V. Mean and variance values for the first two runs of the chemical
independent of adjustment: An explanation and extension of Taguchi’s
process.
signal to noise ratio,” Technometrics, vol. 29, pp. 253-285, 1987.
Experimental [4] G. E. P. Box. “Signal-to-noise ratios, performance criteria, and
Data Mean Variance SNRT
run transformations,” Technometrics, vol. 30, issue 1, pp. 1-17, 1988.
1 57.81 37.29 42.87 47.07 46.26 75.34 14.53 [5] V. N. Nair. “Taguchi’s parameter design: A panel discussion,”
2 24.89 4.35 8.23 14.69 13.04 80.60 3.24 Technometrics, vol. 34, pp. 127-161, 1992.
[6] R. V. León. “Theory of Performance Measures in Robust Design,”
We compare the signal-to-noise ratios of the first two Statistica Sinica, issue 2, pp. 335-358, 1992.
experimental runs. Conducting the required calculations leads [7] M. S. Phadke, Quality Engineering Using Robust Design. Prentice-Hall,
New Jersey, 1989.
to the results summarized in table VI. [8] R. N. Kacker. “Off-line quality control, parameter design, and the
Taguchi method,” Journal of Quality Technology, vol. 17, pp. 176-209,
TABLE VI. Results for the test H 0 : SNRT = SNRT .1 2
1985.
[9] DasGupta, Asymptotic Theory of Statistics and Probability. Springer,
tα 2008.
SNRT1 SNR T2 σ SNR T1 − SNRT2 t
2
,ν
[10] A. Bizimana, “Diseño robusto aplicado a la experimentación agrícola y a
14.53 3.24 2.67 4.23 2.45 procesos agro-industriales,” Ph.D. dissertation, Colegio de
Postgraduados, Mexico, 2015.
[11] E. L. Lehman and J. P. Romano, Testing Statistical Hypotheses,
As t = 4.23 > tα = 2.45 , we conclude that SNRT1 and Springer, 3rd edition, 2005.
,ν
2 [12] G. Casella and R. L. Berger, Statistical Inference, Thomson Learning,
2nd edition, 2002.
SNRT2 are statistically different at the level of significance
[13] J. S. Lawson, Design and Analysis of Experiments with SAS, Chapman
α = 0.05 . & Hall, 2010.
[14] A. Bizimana, J. D. Domínguez and H. Vaquera H., “Robust design of
VII. CONCLUSIONS experiments: A methodological approach,” International Journal of
Modern Engineering Research, vol. 6, issue 11, pp. 01-13, 2016.
This paper presents the statistical tests for pairwise [15] M. Chang, Adaptive Design Theory and Implementation Using SAS and
comparisons of signal-to-noise ratios when the response R, Chapman & Hall, 2008
variable is the nominal the best case. Based on multivariate
delta theorem, the asymptotic distribution of the estimate of

152

A. Bizimana, J. D. Domínguez, and H. Vaquera, “Statistical tests for pairwise comparisons of signal-to-noise ratios: The nominal the best
case,” International Research Journal of Advanced Engineering and Science, Volume 2, Issue 1, pp. 148-152, 2017.