Multidimensional scaling

Multidimensional scaling (MDS) is a

means of visualizing the level of
similarity of individual cases of a
dataset. MDS is used to translate
"information about the pairwise
'distances' among a set of objects or
individuals" into a configuration of
points mapped into an abstract
Cartesian space.[1]

More technically, MDS refers to a set

of related ordination techniques used in
information visualization, in particular
to display the information contained in
a distance matrix. It is a form of non-
linear dimensionality reduction.

Given a distance matrix with the An example of classical multidimensional scaling applied to voting
distances between each pair of objects patterns in the United States House of Representatives. Each red
in a set, and a chosen number of dot represents one Republican member of the House, and each blue
dimensions, N, an MDS algorithm dot one Democrat.
places each object into N-dimensional
space (a lower-dimensional
representation) such that the between-object distances are preserved as well as possible. For N = 1, 2, and
3, the resulting points can be visualized on a scatter plot.[2]

Core theoretical contributions to MDS were made by James O. Ramsay of McGill University, who is also
regarded as the founder of functional data analysis.[3]

Types
MDS algorithms fall into a taxonomy, depending on the meaning of the input matrix:

Classical multidimensional scaling

It is also known as Principal Coordinates Analysis (PCoA), Torgerson Scaling or Torgerson–Gower

scaling. It takes an input matrix giving dissimilarities between pairs of items and outputs a coordinate matrix
whose configuration minimizes a loss function called strain,[2] which is given by

where denote vectors in N-dimensional space, denotes the scalar product between and , and
are the elements of the matrix defined on step 2 of the following algorithm, which are computed from
the distances.

Steps of a Classical MDS algorithm:

Classical MDS uses the fact that the coordinate matrix can be derived by eigenvalue
decomposition from . And the matrix can be computed from proximity matrix
by using double centering.[4]

1. Set up the squared proximity matrix

2. Apply double centering: using the centering matrix ,
where is the number of objects, is the identity matrix, and is an
matrix of all ones.
3. Determine the largest eigenvalues and corresponding eigenvectors
of (where is the number of dimensions desired for the output).
4. Now, , where is the matrix of eigenvectors and is the diagonal
matrix of eigenvalues of .
Classical MDS assumes Euclidean distances. So this is not applicable for direct
dissimilarity ratings.

Metric multidimensional scaling (mMDS)

It is a superset of classical MDS that generalizes the optimization procedure to a variety of loss functions
and input matrices of known distances with weights and so on. A useful loss function in this context is
called stress, which is often minimized using a procedure called stress majorization. Metric MDS minimizes
the cost function called “stress” which is a residual sum of squares:

Metric scaling uses a power transformation with a user-controlled exponent : and for distance.
In classical scaling Non-metric scaling is defined by the use of isotonic regression to
nonparametrically estimate a transformation of the dissimilarities.

Non-metric multidimensional scaling (NMDS)

In contrast to metric MDS, non-metric MDS finds both a non-parametric monotonic relationship between
the dissimilarities in the item-item matrix and the Euclidean distances between items, and the location of
each item in the low-dimensional space. The relationship is typically found using isotonic regression: let
denote the vector of proximities, a monotonic transformation of , and the point distances; then
coordinates have to be found, that minimize the so-called stress,
A few variants of this cost function exist. MDS programs automatically minimize stress in order to obtain
the MDS solution.

The core of a non-metric MDS algorithm is a twofold optimization process. First the optimal monotonic
transformation of the proximities has to be found. Secondly, the points of a configuration have to be
optimally arranged, so that their distances match the scaled proximities as closely as possible. The basic
steps in a non-metric MDS algorithm are:

1. Find a random configuration of points, e. g. by sampling from a normal distribution.

2. Calculate the distances d between the points.
3. Find the optimal monotonic transformation of the proximities, in order to obtain
optimally scaled data .
4. Minimize the stress between the optimally scaled data and the distances by finding a
new configuration of points.
5. Compare the stress to some criterion. If the stress is small enough then exit the
algorithm else return to 2.

Louis Guttman's smallest space analysis (SSA) is an example of a non-metric MDS procedure.

Generalized multidimensional scaling (GMD)

An extension of metric multidimensional scaling, in which the target space is an arbitrary smooth non-
Euclidean space. In cases where the dissimilarities are distances on a surface and the target space is another
surface, GMDS allows finding the minimum-distortion embedding of one surface into another.[5]

Details
The data to be analyzed is a collection of objects (colors, faces, stocks,  .  .  .) on which a distance
function is defined,

distance between -th and -th objects.

These distances are the entries of the dissimilarity matrix

The goal of MDS is, given , to find vectors such that

for all ,

where is a vector norm. In classical MDS, this norm is the Euclidean distance, but, in a broader sense,
it may be a metric or arbitrary distance function.[6]

In other words, MDS attempts to find a mapping from the objects into such that distances are
preserved. If the dimension is chosen to be 2 or 3, we may plot the vectors to obtain a visualization of
the similarities between the objects. Note that the vectors are not unique: With the Euclidean
distance, they may be arbitrarily translated, rotated, and reflected, since these transformations do not change
the pairwise distances .

(Note: The symbol indicates the set of real numbers, and the notation refers to the Cartesian product
of copies of , which is an -dimensional vector space over the field of the real numbers.)

There are various approaches to determining the vectors . Usually, MDS is formulated as an optimization
problem, where is found as a minimizer of some cost function, for example,

A solution may then be found by numerical optimization techniques. For some particularly chosen cost
functions, minimizers can be stated analytically in terms of matrix eigendecompositions.[2]

Procedure
There are several steps in conducting MDS research:

1. Formulating the problem – What variables do you want to compare? How many variables
do you want to compare? What purpose is the study to be used for?
2. Obtaining input data – For example, :- Respondents are asked a series of questions. For
each product pair, they are asked to rate similarity (usually on a 7-point Likert scale from very
similar to very dissimilar). The first question could be for Coke/Pepsi for example, the next
for Coke/Hires rootbeer, the next for Pepsi/Dr Pepper, the next for Dr Pepper/Hires rootbeer,
etc. The number of questions is a function of the number of brands and can be calculated as
where Q is the number of questions and N is the number of brands. This
approach is referred to as the “Perception data : direct approach”. There are two other
approaches. There is the “Perception data : derived approach” in which products are
decomposed into attributes that are rated on a semantic differential scale. The other is the
“Preference data approach” in which respondents are asked their preference rather than
similarity.
3. Running the MDS statistical program – Software for running the procedure is available in
many statistical software packages. Often there is a choice between Metric MDS (which
deals with interval or ratio level data), and Nonmetric MDS[7] (which deals with ordinal data).
4. Decide number of dimensions – The researcher must decide on the number of
dimensions they want the computer to create. Interpretability of the MDS solution is often
important, and lower dimensional solutions will typically be easier to interpret and visualize.
However, dimension selection is also an issue of balancing underfitting and overfitting.
Lower dimensional solutions may underfit by leaving out important dimensions of the
dissimilarity data. Higher dimensional solutions may overfit to noise in the dissimilarity
measurements. Model selection tools like AIC, BIC, Bayes factors, or cross-validation can
thus be useful to select the dimensionality that balances underfitting and overfitting.
5. Mapping the results and defining the dimensions – The statistical program (or a related
module) will map the results. The map will plot each product (usually in two-dimensional
space). The proximity of products to each other indicate either how similar they are or how
preferred they are, depending on which approach was used. How the dimensions of the
embedding actually correspond to dimensions of system behavior, however, are not
necessarily obvious. Here, a subjective judgment about the correspondence can be made
(see perceptual mapping).
6. Test the results for reliability and validity – Compute R-squared to determine what
proportion of variance of the scaled data can be accounted for by the MDS procedure. An R-
 square of 0.6 is considered the minimum acceptable level. An R-square of 0.8 is considered
good for metric scaling and .9 is considered good for non-metric scaling. Other possible tests
are Kruskal’s Stress, split data tests, data stability tests (i.e., eliminating one brand), and test-
retest reliability.
7. Report the results comprehensively – Along with the mapping, at least distance measure
(e.g., Sorenson index, Jaccard index) and reliability (e.g., stress value) should be given. It is
also very advisable to give the algorithm (e.g., Kruskal, Mather), which is often defined by
the program used (sometimes replacing the algorithm report), if you have given a start
configuration or had a random choice, the number of runs, the assessment of dimensionality,
the Monte Carlo method results, the number of iterations, the assessment of stability, and the
proportional variance of each axis (r-square).

Implementations
ELKI includes two MDS implementations.
MATLAB includes two MDS implementations (for classical (cmdscale) and non-classical
(mdscale) MDS respectively).
The R programming language offers several MDS implementations, e.g. base cmdscale
function, packages smacof (https://CRAN.R-project.org/package=smacof)[8] (mMDS and
nMDS), and vegan (https://CRAN.R-project.org/package=vegan) (weighted MDS).
scikit-learn contains function sklearn.manifold.MDS (http://scikit-learn.org/stable/modules/ge
nerated/sklearn.manifold.MDS.html).

See also
Data clustering
Factor analysis
Discriminant analysis
Dimensionality reduction
Distance geometry
Cayley–Menger determinant
Sammon mapping
Iconography of correlations

References
1. Mead, A (1992). "Review of the Development of Multidimensional Scaling Methods". Journal
of the Royal Statistical Society. Series D (The Statistician). 41 (1): 27–39. JSTOR 2348634
(https://www.jstor.org/stable/2348634). "Abstract. Multidimensional scaling methods are now
a common statistical tool in psychophysics and sensory analysis. The development of these
methods is charted, from the original research of Torgerson (metric scaling), Shepard and
Kruskal (non-metric scaling) through individual differences scaling and the maximum
likelihood methods proposed by Ramsay."
2. Borg, I.; Groenen, P. (2005). Modern Multidimensional Scaling: theory and applications
(2nd ed.). New York: Springer-Verlag. pp. 207–212. ISBN 978-0-387-94845-4.
3. Genest, Christian; Nešlehová, Johanna G.; Ramsay, James O. (2014). "A Conversation with
James O. Ramsay" (https://www.jstor.org/stable/43299752). International Statistical Review /
Revue Internationale de Statistique. 82 (2): 161–183. JSTOR 43299752 (https://www.jstor.or
g/stable/43299752). Retrieved 30 June 2021.
 4. Wickelmaier, Florian. "An introduction to MDS." Sound Quality Research Unit, Aalborg
University, Denmark (2003): 46
5. Bronstein AM, Bronstein MM, Kimmel R (January 2006). "Generalized multidimensional
scaling: a framework for isometry-invariant partial surface matching" (https://www.ncbi.nlm.ni
h.gov/pmc/articles/PMC1360551). Proc. Natl. Acad. Sci. U.S.A. 103 (5): 1168–72.
Bibcode:2006PNAS..103.1168B (https://ui.adsabs.harvard.edu/abs/2006PNAS..103.1168
B). doi:10.1073/pnas.0508601103 (https://doi.org/10.1073%2Fpnas.0508601103).
PMC 1360551 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1360551). PMID 16432211
(https://pubmed.ncbi.nlm.nih.gov/16432211).
6. Kruskal, J. B., and Wish, M. (1978), Multidimensional Scaling, Sage University Paper series
on Quantitative Application in the Social Sciences, 07-011. Beverly Hills and London: Sage
Publications.
7. Kruskal, J. B. (1964). "Multidimensional scaling by optimizing goodness of fit to a nonmetric
hypothesis". Psychometrika. 29 (1): 1–27. doi:10.1007/BF02289565 (https://doi.org/10.100
7%2FBF02289565). S2CID 48165675 (https://api.semanticscholar.org/CorpusID:4816567
5).
8. Leeuw, Jan de; Mair, Patrick (2009). "Multidimensional Scaling Using Majorization:
SMACOF in R" (http://www.jstatsoft.org/v31/i03/). Journal of Statistical Software. 31 (3).
doi:10.18637/jss.v031.i03 (https://doi.org/10.18637%2Fjss.v031.i03). ISSN 1548-7660 (http
s://www.worldcat.org/issn/1548-7660).

Bibliography
Cox, T.F.; Cox, M.A.A. (2001). Multidimensional Scaling. Chapman and Hall.
Coxon, Anthony P.M. (1982). The User's Guide to Multidimensional Scaling. With special
reference to the MDS(X) library of Computer Programs. London: Heinemann Educational
Books.
Green, P. (January 1975). "Marketing applications of MDS: Assessment and outlook".
Journal of Marketing. 39 (1): 24–31. doi:10.2307/1250799 (https://doi.org/10.2307%2F12507
99). JSTOR 1250799 (https://www.jstor.org/stable/1250799).
McCune, B. & Grace, J.B. (2002). Analysis of Ecological Communities. Oregon, Gleneden
Beach: MjM Software Design. ISBN 978-0-9721290-0-8.
Young, Forrest W. (1987). Multidimensional scaling: History, theory, and applications.
Lawrence Erlbaum Associates. ISBN 978-0898596632.
Torgerson, Warren S. (1958). Theory & Methods of Scaling. New York: Wiley. ISBN 978-0-
89874-722-5.

Retrieved from "https://en.wikipedia.org/w/index.php?title=Multidimensional_scaling&oldid=1157398985"