Received: 16 June 2021 Revised: 20 June 2021 Accepted: 24 June 2021

DOI: 10.1002/sono.12276

EDUCATION ARTICLE

Correlation does not imply agreement: A cautionary tale for

researchers and reviewers

Christopher Edwards1,2,3,4 | Heather Allen1,5 | Crispen Chamunyonga1,4

1
School of Clinical Sciences, Faculty of Health,
Queensland University of Technology, Abstract
Brisbane, Queensland, Australia Sonography researchers are often confronted with questions concerning the relation-
2
Mater Research Institute-University of
ship between measurements acquired in clinical practice. This educational commen-
Queensland, South Brisbane, Queensland,
Australia tary provides clarity on the definition of the terms correlation and agreement. We
3
Department of Medical Imaging, Redcliffe discuss the statistical tests used to assess correlation and outline some common pit-
Hospital, Redcliffe, Queensland, Australia
4
falls authors fall into when reporting these. We provide examples of the inaccurate
Centre for Biomedical Technologies,
Queensland University of Technology, use of these tests in the sonography literature and recommend better alternatives for
Brisbane, Queensland, Australia
assessing agreement. This review will benefit authors embarking on studies compar-
5
Department of Medical Imaging, The Prince
Charles Hospital, Chermside, Queensland,
ing measurements and reviewers considering manuscripts to ensure the methodology
Australia used is sufficient to justify the claims. The authors also provide plots created in the R

Correspondence
statistical software package, and the supplementary data file supplied can be used
Christopher Edwards, School of Clinical and adapted for those wishing to perform similar statistical tests.
Sciences, Queensland University of
Technology, Gardens Point Campus, 2 George
KEYWORDS
St, Brisbane, QLD 4000, Australia.
Email: c8.edwards@qut.edu.au agreement, biostatistics, Bland–Altman plot, correlation, intra-class correlation

1 | I N T RO DU CT I O N methods used in the context of continuous variables. This discus-

Research in clinical ultrasound practice often involves demonstrating
an association between numerical observations and measurements.
These may be direct clinical characteristics such as investigating a
relationship between age and renal length measured by ultrasound, or
in the context of a department audit, investigating the number of
scans performed as they relate to years of experience. Studies are also
designed to test whether measurements are reproducible under dif-
ferent conditions or if there is agreement between a single measure-
ment acquired using different technologies. An example, which we
will return to later in this article, is the comparison of transabdominal
ovarian volume measurements with measurements obtained using a
transvaginal approach. All observations from these groups can be
measured as numerical values and are referred to as continuous vari-
ables.1 When researchers are interested in reporting the degree of
association or concordances between such continuous variables, cor-
relation and agreement are often used. However, the term ‘correla-
tion’ as a synonym for ‘agreement’ can be misleading in research.
This educational article aims to clarify the terms correlation
and agreement and explore some of the standard statistical
sion and the tests described will aid sonographers and reviewers as
they conduct and appraise research to reduce inconsistencies and
improve quality.
2 | WHAT IS CORRELATION?
In mathematical terms, correlation is a test that summarises the
strength and direction of a relationship between two independent
continuous variables. In most cases, it refers to a linear relationship,
expressed as the Pearson correlation coefficient (r), and has a value
ranging from 1 to +1.4 A positive correlation occurs when the
value of one continuous variable increases in tandem with the other.
In contrast, a negative correlation occurs when, as one variable
increases, the other decreases. A coefficient of 1 represents a perfect
linear relationship, meaning all values fit perfectly to a straight line. In
contrast, a coefficient of 0 or close to zero indicates the lack of any
2,3
linear relationship. The units of each variable do not influence the
value of the Pearson correlation coefficient. Other types of coeffi-
cients are available for non-linear data, such as Spearman5 and the

Sonography. 2021;1–6. wileyonlinelibrary.com/journal/sono © 2021 Australasian Sonographers Association 1

2 EDWARDS ET AL.
Kendall rank correlation test.6 This discussion will concentrate on data
with a linear relationship and refer to the Pearson coefficient.
Notably, the magnitude of the correlation coefficient only pro-
vides information on how close the points lie from the regression or
‘least-squares line’. The position of this line is calculated mathemati-
cally and drawn as a straight line that minimises the sum of the
7
squared errors, referred to as least squares regression. The coeffi-
cient does not provide any information on the absolute agreement
between two variables. In most scenarios, two sets of variables are
never perfectly correlated (r = 1), and there remains an error between
each data point and the regression line. Figure 1 demonstrates four
Pearson coefficient values. In plots A–C, the slope and direction of
each corresponding regression line remain relatively unchanged; how-
ever, the distance between the data points and the regression line
increases, resulting in a lower coefficient value. The coefficient calcu-
lation also ignores the units of each variable, and reversing the graph
axes below will not affect the value of r.
Before interpreting a correlation coefficient, it is essential to plot
the relationship between the variables of interest on a scatter plot to
avoid incorrect assumptions. This is the easiest way to find the nature
of the correlation. Inspecting the data in this way reveals whether the
correlation is linear or non-linear. In some instances, data may be
highly correlated but related in a non-linear monotonic fashion. For
example, when the increase or decrease of one variable occurs at a
different rate to the other. The plotted data in this instance would
appear curved (or skewed), and alternative correlation tests are more
appropriate. The Pearson correlation is also highly influenced by the
presence of outliers or subgroups, or if significant clustering of one
F I G U R E 1 Scatter plots to demonstrate the relationship between two sets of variables. Figure 1A–C shows a positive linear correlation,
indicated by Pearson coefficient values. In Figure 1D, there is no relationship
variable (range restriction) is present; again, visual inspection of the
data will reveal these.8 Although Pearson's correlation does not
assume normality, it does assume constant variance (homoscedastic-
ity). Unequal variance (heteroscedasticity) occurs in situations where
there is a difference of dispersion of one variable across the range of
values of the other. Graphical examples of cases in which the Pearson
correlation should not be used are shown in Figure 2 below.
Correlation analysis also assumes that all observations are inde-
pendent of each other. Independent variables are those in which the
distribution of one variable does not depend on the other. Therefore,
it is inappropriate to use in scenarios involving repeated measure-
ments. Some examples of non-independent studies may be; compar-
ing the peak systolic velocity in an artery before and after exercise, or
from the earlier example, comparing two different ultrasound tech-
niques (transabdominal vs. transvaginal). In these examples, a repeated
measurement is acquired from a single patient. Unfortunately, this is a
common misuse of the correlation coefficient in the medical literature,
particularly in studies that aim to assess ‘agreement’ rather than
‘correlation’.
A correlation coefficient is fairly easy to calculate in most statisti-
cal software packages such as the R statistical package9 (https://
www.r-project.org/) or IBM's SPSS Statistics Software (IBM Corp.
Armonk, NY). It can be computed to compare two variables or create
a matrix to quickly assess how one variable is correlated to a range of
others, which is helpful to select relevant variables for further regres-
sion analysis. The calculation is an example of pairwise testing, mean-
ing that a value for each variable is required; as such, in most cases,
the software will automatically ignore missing values.
EDWARDS ET AL.
F I G U R E 2 Examples where the Pearson correlation coefficient should not be applied. (A) Data containing outliers, (B) heteroscedasticity, in
which there is unequal variance between the two variables, (C) clustered or grouped data (D), non-linear or curved relationships
3 | S T A T I S T I C A L S I G N I F I C A N C E A N D R-
SQ UARED
Most statistical software reports a corresponding p-value when calculat-
ing the correlation coefficient. In this instance, the p-value represents the
chance that a significant linear correlation does not exist between the
variables (the null hypothesis). The value of p will depend on the number
of observations, and a low p-value (p ≤0.05) provides confidence that the
measured correlation coefficient in the sample represents a ‘real’ correla-
tion between the two variables in the wider population. However, a low
p-value does not provide further evidence on the degree of correlation.
Even medium sample sizes with small correlation coefficients can show p
values that are highly significant and hence should not be used out of
context. In particular, statistical significance should not be confused with
clinical significance. The p-value only provides a measure of the test's
statistical reliability. The analysis of confidence intervals and study design
provides complementary evidence of clinical significance.10
2
r-squared (r ), the square of the correlation coefficient, is a related
measure often encountered in linear regression analysis. When a cor-
relation exists, linear regression enables the calculation of the equa-
tion that minimises the distance between the fitted line and all the
data points in the sample. This is useful to develop a model for
the prediction of one value when another value is known. For exam-
ple, predicting an individual's weight from their measured height. In
simple linear regression, the r-squared provides a percentage of how
well the model fits the data, sometimes referred to as the ‘goodness
of fit’. In technical terms, r-squared indicates the percentage of vari-
ance in the dependant variable (weight) that the independent variable
3
(height) explains and attempts to answer the question; Does the
model do a good job explaining changes in the dependent variable?
Several assumptions need to be met and tested during linear regres-
sion analysis, which is beyond the scope of this article; however, it
provides a distinction and relationship between r and r-squared.
4 | WHAT IS AGREEMENT?
The term agreement is separate and represents the concordance
between two measurements of one variable. In other words, it is used
to assess whether measurements by two observers or two different
techniques yield similar results.2 We can explore this concept using the
earlier example of comparing ovarian volumes measured via trans-
abdominal and transvaginal techniques. In this example, the actual vol-
ume of the ovary is unknown, and the analysis should be designed to
assess how closely measurements are taken from one method match
those of another, not simply their relationship. Since we are measuring
the same structure on the same patient (albeit with a different tech-
nique), the two values will be highly correlated but may not necessarily
agree (agreement could be by chance or error). In this example,
Pearson's or similar correlation coefficient should not be applied. To
demonstrate this, we provide two additional plots (Figure 3), using fic-
tional data. Transvaginal scan (TVS) measurements have been plotted
on the x-axis and Transabdominal scan (TAS) measurements on the y-
axis. In both examples, the calculated correlation coefficient remains
high (r ≈ 0.9), and it may be tempting to state that the two methods
agree, but by inspecting the relationship on a scatter plot, it is easy to
4 EDWARDS ET AL.
F I G U R E 3 Scatter plots to demonstrate the relationship between two highly correlated imaging methods with bias. (A) Systematic bias and
(B) proportional bias
see that both demonstrate poor agreement with significant bias. In the
first example (Figure 3A), TAS consistently measures the structures
higher, a systematic bias. In Figure 2B, TAS overestimates the value
when levels are low and underestimate when values are high compared
to TVS. The dashed line represents perfect agreement in both plots.
To highlight some similar examples from the literature, in a study
by Riesta-Candelaria et al.,11 the authors sought to evaluate ultrasound
measurements of right liver lengths using different ultrasound modes.
As part of the study, Pearson's correlation coefficient was used to com-
pare standard 2D and panoramic imaging measurements in different
imaging planes. The study concluded that the craniocaudal plane is
optimal for performing these measurements, justified by the highest r-
value. From the previous discussion, it should be evident that measure-
ments of the same liver, regardless of the imaging method, are likely to
be highly correlated. Furthermore, r values approaching 1 do not pro-
vide sufficient evidence that the two methods agree. Indeed, a high
r-value may indicate almost perfect agreement, but without further
analysis to assess the presence of bias, the agreement remains
unknown. In another example, Lam et al.12 studied an image review
scoring system to assess sonographer performance during morphology
examinations. As part of the study, individual scores from two
reviewers were graphed as scatter plots and compared using Pearson's
correlation coefficient. An increase in the correlation coefficient
between the pre- and post-training phase of the study was used to
demonstrate the utility of the scoring system. Once again, the correla-
tion coefficient has been incorrectly applied to assess agreement. The
same score (or close to) used for both reviewers for each image set
would imply good agreement. However, one reviewer may consistently
mark each image set three points higher than another reviewer,
producing an almost perfect correlation but poor agreement.
5 | ALTERNATIVE METHODS
Two common alternative methods to assess the agreement between
continuous measurements are available: the intra-class correlation
coefficient (ICC)13 and the Bland–Altman plot with limits of agree-
ment.14 The ICC provides a single measure of the agreement between
two pairs of measurements.13 In contrast, a Bland–Altman plot offers
a graphical alternative to a scatter plot to assess both the agreement
and bias between paired measurements.14 Methods used to assess
agreement between measurements vary depending on the type of
variable (continuous or categorical) and the number of observers.
Since the focus of this article is continuous variables, we limit the dis-
cussion to two common methods used for this type of variable. Ordi-
nary least squared regression could potentially be used to assess
agreement; however, this type of simple regression assumes one
method is error-free and is unrealistic in most settings. Alternatively,
other types of regression are better suited for method comparison
studies, such as Deming's linear regression15 or Passing Bablok linear
regression,16 which readers may wish to explore.17
5.1 | Intra-class correlation coefficient
The ICC is often encountered in studies designed to assess measure-
ment variability caused by the influence of individual operators.18 For
example, a study evaluating the variability of measured left ventricular
ejection fractions under different circumstances.19 The ICC provides a
single measure of the extent to which the readings agree between the
operators, instruments, or time points. It conveys the between pair
variances as a proportion of the total variance in a group of observa-
tions.20 The ICC is expressed as a unitless number from 0 to 1, with
zero indicating no agreement and 1 indicating perfect agreement.
Although no strict guidelines are available, and it depends on the clini-
cal context of the observations, values greater than 0.9 are often cited
as excellent agreement, between 0.75 and 0.9 as good, between 0.5
and 0.75 moderate and below 0.5 as poor agreement.21,22 One advan-
tage of the ICC is that more than two sets of measurements can be
assigned to a single ICC value. For example, the agreement between
measurements performed by four sonographers or using three differ-
ent ultrasound machines. Several different forms of ICCs are available,
and both authors and reviewers should be aware that the selection of
ICC should be consistent with the stated aims of any research project.
In addition, when reporting ICC values, including the 95% confidence
interval is essential to measure the accuracy of the calculation. For
EDWARDS ET AL.
F I G U R E 4 Bland–Altman plots showing agreement between two sets of measurements. The solid line shows the mean difference between
the values while the dotted lines are the upper and the lower limits of agreement (95% observed differences)
more information on selecting and reporting ICCs, readers are
encouraged to consult the excellent guide by Koo et al.22
5.2 | Bland–Altman plots
When the agreement between two measurements is required, a
Bland–Altman Plot provides a convenient process of comparing two
techniques or methods.14 A Bland–Altman plot can also be used in
combination with the ICC and is a scatter plot of the difference
between two measurements on the Y-axis (Method 1 minus Method 2)
against the mean of the two measurements on the X-axis ({Method 1
+ Method 2}/2). This type of basic plot reveals any systematic differ-
ences (bias) between measurements, which is the difference between
Method 1 and Method 2. On the same plot, the limits of agreement
(LOA) are plotted as separate lines, demonstrating the range within
which 95% of the differences between one method and the other are
included.23 The LOA are expressed as the mean observed difference
±1.96 Standard Deviations (SD) of observed differences.2 The interpre-
tation of what is acceptable to the limits of the agreement will depend
on the clinical context; however, broad LOA or values that consistently
fall outside these bounds likely indicates a lack of agreement between
the two methods. To provide further context on how a Bland–Altman
plot is displayed and interpreted, plots of the same values used in
Figures 1A,C and 2A,B are shown in Figure 4 below.
5
6 | CONC LU SION
The terms correlation and agreement are not interchangeable in
the scientific literature, and sonographers and reviewers must
understand their differences. In essence, correlation can be used to
infer an association between different independent variables. On
the other hand, agreement is the concordance between measure-
ments or techniques for the same variable. The Pearson's correla-
tion coefficient and its associated p-value are commonly cited for
inferring an association, but two highly correlated variables may
not agree. This article outlines the situations in which correlation
may be used and offers better alternatives for studies that aim to
assess agreement.
CONFLIC T OF INT ER E ST
Christopher Edwards is an editorial board member for Sonography
and a co-author on this article. CE was blinded and not involved in the
peer review process; management of the peer review process and
decision-making for this article was undertaken by the Editor-in-Chief,
Kerry Thoirs, acting as Handling Editor.
OR CID
Christopher Edwards https://orcid.org/0000-0001-7466-9530
Heather Allen https://orcid.org/0000-0002-4680-4149
Crispen Chamunyonga https://orcid.org/0000-0002-6714-4362
6 EDWARDS ET AL.
RE FE R ENC E S
1. Dettori JR, Norvell DC. The anatomy of data. Glob Spine J. 2018;8:
311–3. https://doi.org/10.1177/2192568217746998
2. Ranganathan P, Pramesh CC, Aggarwal R. Common pitfalls in statisti-
cal analysis: measures of agreement. Perspect Clin Res. 2017;8:187–
91. https://doi.org/10.4103/picr.PICR_123_17
3. Rosso A. Correlation does not mean agreement: why is it still used as
a synonym of agreement? Radiology. 2015;276:617–9. https://doi.
org/10.1148/radiol.2015150302
4. Pearson K, Lee A. On the laws of inheritance in man: I. Inheritance of
physical characters. Biometrika. 1903;2:357. https://doi.org/10.
2307/2331507
5. Dodge Y. Spearman rank correlation coefficient. The concise encyclo-
pedia of statistics. New York, NY: Springer; 2008. p. 502–5.
6. George M. Kendall rank correlation coefficient. The concise encyclo-
pedia of statistics. Volume 1938. New York, NY: Springer; 2008.
p. 278–81.
7. Aggarwal R, Ranganathan P. Common pitfalls in statistical analysis: lin-
ear regression analysis. Perspect Clin Res. 2017;8:100–2. https://doi.
org/10.4103/2229-3485.203040
8. Armstrong RA. Should Pearson's correlation coefficient be avoided?
Ophthalmic Physiol Opt. 2019;39:316–27. https://doi.org/10.1111/
opo.12636
9. R Core Team R: a language and environment for statistical computing;
2013.
10. du Prel J-B, Hommel G, Röhrig B, Blettner M. Confidence interval or
P-value? Part 4 of a series on evaluation of scientific publications.
Dtsch Aerzteblatt. 2009;106:335–9. https://doi.org/10.3238/arztebl.
2009.0335
11. Riestra-Candelaria BL, Rodriguez-Mojica W, Jorge JC. Anatomical
criteria to measure the adult right liver lobe by ultrasound. Sonogra-
phy. 2018;5:181–6. https://doi.org/10.1002/sono.12162
12. Lam P, Samson A, Magotti R, Benzie R. The effect of preliminary
training on quantitative evaluation of sonographer performance in
the fetal morphology ultrasound examination. Australas J Ultrasound
Med. 2013;16:142–6. https://doi.org/10.1002/j.2205-0140.2013.
tb00102.x
13. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater
reliability. Psychol Bull. 1979;86:420–8. https://doi.org/10.1037/
0033-2909.86.2.420
14. Martin Bland J, Altman D. Statistical methods for assessing agree-
ment between two methods of clinical measurements. Lancet. 1986;
327:307–10. https://doi.org/10.1016/S0140-6736(86)90837-8
15. Deming WE. Statistical adjustment of data. Oxford, England: Wiley;
1943.
16. Passing H, Bablok W. A new biometrical procedure for testing the
equality of measurements from two different analytical methods.
Application of linear regression procedures for method comparison
studies in clinical chemistry, part I. Clin Chem Lab Med. 1983;21:
709–20. https://doi.org/10.1515/cclm.1983.21.11.709
17. Twomey PJ, Kroll MH. How to use linear regression and correlation
in quantitative method comparison studies. Int J Clin Pract. 2008;62:
529–38. https://doi.org/10.1111/j.1742-1241.2008.01709.x
18. Popovic ZB, Thomas JD. Assessing observer variability: a user's guide.
Cardiovasc. Diagn. Ther. 2017;7:317–24. https://doi.org/10.21037/
cdt.2017.03.12
19. Kusunose K, Shibayama K, Iwano H, Izumo M, Kagiyama N,
Kurosawa K, et al. Reduced variability of visual left ventricular ejection
fraction assessment with reference images: the Japanese Association
of Young Echocardiography Fellows multicenter study. J Cardiol. 2018;
72:74–80. https://doi.org/10.1016/j.jjcc.2018.01.007
20. Weir JP. Quantifying test-retest reliability using the intraclass correla-
tion coefficient and the SEM. J Strength Cond Res. 2005;19:231–40.
https://doi.org/10.1519/15184.1
21. Portney LG, Watkins MP. Foundations of clinical research: applica-
tions to practice. Vol 892. Upper Saddle River, NJ: Pearson/Prentice
Hall; 2009.
22. Koo TK, Li MY. A guideline of selecting and reporting Intraclass corre-
lation coefficients for reliability research. J Chiropr Med. 2016;15:
155–63. https://doi.org/10.1016/j.jcm.2016.02.012
23. Van Stralen KJ, Jager KJ, Zoccali C, Dekker FW. Agreement between
methods. Kidney Int. 2008;74:1116–20. https://doi.org/10.1038/ki.
2008.306
SUPPORTING INF ORMATION
Additional supporting information may be found online in the
Supporting Information section at the end of this article.
How to cite this article: Edwards C, Allen H, Chamunyonga C.
Correlation does not imply agreement: A cautionary tale for
researchers and reviewers. Sonography. 2021;1–6. https://
doi.org/10.1002/sono.12276