See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/301516298

Multivariate Analysis : An Overview

Article · September 2013

CITATIONS READS
6 29,743

1 author:

Siddharth Kumar Singh

Saraswati Dental College and Hospital
41 PUBLICATIONS   269 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Research work View project

All content following this page was uploaded by Siddharth Kumar Singh on 20 April 2016.

The user has requested enhancement of the downloaded file.

REVIEW ARTICLE

Multivariate Analysis : An Overview

Sandeep Kumar*, Siddharth Kumar Singh*, Prashant Mishra*

Abstract
Introduction: Multivariate analysis (MVA) techniques allow more than two variables to be
analysed at once. Two general types of MVA technique: Analysis of dependence& Analysis of
interdependence. Technique is selected depending on type of data and reason for the analysis.
Cluster analysis: “Techniques for identifying separate groups of similar cases”. Also used to
summarize data by defining segments of similar cases in the data. Discriminant analysis: Is a
statistical technique for classifying individuals or objects into mutually exclusive and
exhaustive groups on the basis of a set of independent variables”. Factor analysis :
Multiple factor analysis (mfa): “Statistical method used to describe variability among
observed variables in terms of a potentially lower number of unobserved variables called
factors”. Correspondance Analysis: “Technique that generates graphical representations of the
interactions between modalities (or "categories") of two categorical variables”. Regression
Analysis: “Refers to any techniques for modelling and analyzing several variables, when the focus
is on the relationship between a dependent variable and one or more independent variables.”
Multiple Linear Regression Analysis (MLR): In Multiple linear regressions, several independent
variables are used to predict with a least square approach one direct variable. Multivariate
analysis of variance (MANOVA): It is a generalized form of univariate analysis of
variance (ANOVA). Conclusion: Because there are many potential problems and pitfalls in the
use of multivariable techniques in clinical research, these procedures should be used with care.
(Kumar S, Singh SK, Mishra P. Multivariate Analysis : An Overview. www.journalofdentofacialsciences.com,
2013; 2(3): 19-26)

Introduction
*Sr Lecturer, Sri Aurbindo College of Dentistry, Indore, “Application of methods that deal with
Madhya Pradesh reasonably large number of measurements made
Address for Correspondence: on each object in one or more samples
**Dr Sandeep Kumar simultaneously”.1 Many statistical techniques focus
Flat No. 304, Sanskar Block, SAIMS Campus, on just one or two variables. Multivariate analysis
Sanwar Road, Indore, Madhya Pradesh
(MVA) techniques allow more than two variables
e-mail: drsandeep40@yahoo.com
to be analysed at once. The ultimate goal of these
analyses is either explanation or prediction, i.e.,
more than just establishing an association.
Multivariate Analysis Methods: Two general
types of MVA technique:1
20 Kumar et al.

• Analysis of dependence: Where one (or more) • Regional analyses—classify cities into
variables are dependent variables, to be typologies based on demographic
explained or predicted by others. e.g. Multiple variables.
regression, Discriminant analysis, Manova, • Marketing research--classifying customers
Partial Least Square. based on product use.
• Analysis of interdependence: No variables • Chemistry—classification of compounds
thought of as “dependent”. Look at the based upon properties
relationships among variables, objects or cases.
Overview of cluster analysis [3]
E.g. cluster analysis, factor analysis, and
principal component analysis. Step 1=n objects measured on p variables
Selection of Technique depends on:1 Step 2=transform into N x N similarity matrix
9 The type of data under analysis: Nominal Step3=cluster formation (Mutually exclusive
data, Ordinal data clusters, Hierarchical clusters
9 The reason for the analysis: Classify data, Step 4=cluster profile
Data reduction Types of Cluster Analysis4
Classifying data ƒ Hierarchical
ƒ Hierarchical cluster analysis ƒ Two Step
ƒ Two step cluster analysis ƒ K Means
ƒ K means cluster analysis Hierarchical cluster: Perform successive fusions
ƒ Discriminant analysis or divisions of the data. Hierarchical clustering is
Data reduction one of the most straightforward methods. It can be
ƒ Factor analysis either agglomerative or divisive.
ƒ Correspondence analysis Agglomerative hierarchical clustering: Every
case being a cluster unto itself. At successive steps,
Level of Measurement and Multivariate similar clusters are merged. Proceeds by forming a
Statistical Technique:2 series of fusions of the n objects into groups
Independent Dependent Technique
Divisive clustering: Starts with everybody in
Variable Variable
one cluster and ends up with everyone in
Numerical Numerical Multiple
Regression individual clusters. It partitions the set of n objects
Nominal or Nominal Logistic into finer and finer subdivisions.
Numerical Regression Agglomerative Methods
Nominal or Numerical Cox Regression ƒ Single linkage or nearest neighbor method
Numerical (censored)
Nominal or Numerical ANOVA, ƒ Complete linkage or the farthest neighbor
Numerical MANOVA method
Nominal or Nominal (2 or Discriminant ƒ Average linkage
Numerical more values) Analysis ƒ Wards error sum of squares methods
Cluster Analysis:3 “Techniques for Divisive Methods
identifying separate groups of similar ƒ A splinter average distance methods.
cases”. Also used to summarize data by defining
ƒ Automatic interaction detection.
segments of similar cases in the data. This use of
cluster analysis is known as “dissection.” The output from both agglomerative and
divisive methods is typically summarized by use of
Application:
a dendrogram.
• Psychology—classifying individuals
according to personality types.

www.journalofdentofacialsciences.com Vol. 2 Issue 3

Kumar et al.
“It is a two dimensional tree like diagram
illustrating the fusion or partition that have been
affected at each successive levels”.
To form clusters using a hierarchical cluster
analysis, the following things need to be
considered: A criterion for determining similarity or
distance between cases, A criterion for determining
which clusters are required to be merged at
successive steps, The number of clusters needed to
represent the data, The hierarchical clustering
procedure attempts to identify relatively
homogeneous groups of cases based on selected
characteristics, using an algorithm that starts with
each case in a separate cluster and combines
clusters until only one is left.
How Many Clusters Needed??
No specific number. It depends on what we
are going to do with them. (Type of Analysis). To
find a good cluster solution: The characteristics of
the clusters at successive steps needs to be
evaluated.
Two Step Cluster Analyses: “A clustering
procedure that can rapidly form clusters on the
basis of either categorical or continuous data.”
Used in large data sets. It requires only one pass of
data which is important for very large data files
and it can produce solutions based on mixtures of
continuous and categorical variables and for
varying numbers of clusters. The clustering
algorithm is based on a distance measure that
gives the best results if all variables are
independent, continuous variables have a normal
distribution, and categorical variables have a
multinomial distribution. Two set cluster analysis is
designed to reveal natural groupings (or clusters)
within a data set that would otherwise not be
apparent.
Steps Involved3
Step 1: Preclustering: Making Little
Clusters i.e. the goal of preclustering is to reduce
www.journalofdentofacialsciences.com
21
the size of the matrix that contains distances
between all possible pairs of cases.
Step 2: Hierarchical Clustering of
Preclusters In the second step, SPSS uses the
standard hierarchical clustering algorithm on the
preclusters. This helps to explore a range of
solutions with different numbers of clusters. We
can select the number of cluster to be formed by
clicking.
• Number of Clusters: Determine
automatically, Specify fixed.
• Two options for categorical and
continuous variables.
Two different criteria are used: Schwarzs
Bayesian Criterion, Akaikes Information Criterion
‰ Cluster formed can then be utilized for
¾ Examining the composition of the clusters.
¾ Examining the importance of individual
variables within the clusters.
¾ Looking at all variables within a cluster
¾ Looking at the relationship to other
variables
K Means cluster analysis: This procedure
attempts to identify relatively homogeneous
groups of cases based on selected characteristics,
using an algorithm that can handle large numbers
of cases. However, the algorithm requires
specifying the number of clusters.
Guidelines on the appropriate use of clustering
methodologies [1]
¾ Outliers should be removed prior to
analysis (sensitive)
¾ The selection of similarity or distance
measure to use is still unanswered.
¾ Care and good judgment should be used
in setting parameter values
¾ Wherever possible data should be split and
cross replicated so as to assess the stability
of cluster solutions.
Discriminant Analysis5
“Is a statistical technique for classifying individuals
or objects into mutually exclusive and exhaustive
groups on the basis of a set of independent
variables”.
Vol. 2 Issue 3
22 Kumar et al.

Types Procedure1
Discrete discriminant analysis ‰ Performed in two steps
Logistic discrimination 9 Select the factors
‰ Descriptive
Error rate estimation
9 Initial solution
• The re-substitution method 9 Coefficient
• The Hold-out method 9 KMO+BARTLETT TEST: Check
• The U method or cross validation Bartlett table, KMO value 0-1(close to
1 better the significance at .05) , Using
• The Jackknife method
kieser criteria factors are selected,
APPLICATION Scree plot can be used for better
• Face identification judgement
• Bankruptcy detection ‰ Extraction
9 Principal component
• Marketing research
9 Correlation matrix
Factor Analysis1 9 Eigen value
Multiple Factor Analysis (MFA): “Statistical 9 Scree plot
method used to describe variability among ‰ Rotation
observed variables in terms of a potentially lower 9 No change
number of unobserved variables called factors”. ‰ Scores
Factor analysis attempts to identify underlying 9 Exclude case listwise
variables, or factors, that explain the pattern of ‰ Descriptive
correlations within a set of observed variables. 9 Untick
Factor analysis is often used in data reduction and ‰ Extraction
can also be used to generate hypotheses regarding 9 Scree plot
causal mechanisms or to screen variables for 9 Univariate untick
subsequent analysis 9 Select the factors
Type of factor analysis ‰ Rotation
9 Varimax-independent variables
Exploratory factor analysis : (EFA) is used to
9 Oblimin dependent variables
uncover the underlying structure of a relatively
large set of variables. The researcher's a NUMBER CALLED “LOADING”. Higher loading
priori assumption is that any indicator may be is selected and close loading is eliminated
associated with any factor. This is the most Types of factoring1
common form of factor analysis. Principal Component Analysis:2 PCA was
Confirmatory factor analysis : (CFA) seeks to invented in 1901 by Karl Pearson. The goal of
determine if the number of factors and the PCA is to decompose a data table with correlated
loadings of measured (indicator) variables on them measurements into a new set of uncorrelated (i.e.,
conform to what is expected on the basis of pre- orthogonal) variables. These variables are called,
established theory. The researcher's à priori depending upon the context, principal
assumption is that each factor is associated with a components, factors, eigenvectors, singular
specified subset of indicator variables vectors, or loadings. The results of the analysis are
Factor analysis is related to principal component often presented with graphs plotting the
analysis (PCA), but the two are not identical. The projections of the units onto the components, and
two methods become essentially equivalent if the the loadings of the variables.
error terms in the factor analysis model can be Canonical factor analysis: It is also called Rao's
assumed to all have the same variance. canonical factoring. Seeks factors which have the
highest canonical correlation with the observed

www.journalofdentofacialsciences.com Vol. 2 Issue 3

Kumar et al.
variables. It is unaffected by arbitrary rescaling of
the data.
Common factor analysis: It is also called
principal factor analysis (PFA) or principal axis
factoring (PAF). Seeks the least number of factors
which can account for the common variance
(correlation) of a set of variables.
Image factoring: Based on the correlation matrix
of predicted variables rather than actual variables,
where each variable is predicted from the others
using multiple regression.
Alpha factoring: Based on maximizing the
reliability of factors, assuming variables are
randomly sampled from a universe of variables.
Correspondance Analysis6
“Technique that generates graphical
representations of the interactions between
modalities (or "categories") of two categorical
variables”. It allows the visual discovery and
interpretation of these interactions, that is, of the
departure from independence of the two variables
Steps6
Run a Chi-square test of independence on
these two variables
• If the test fails to reject the independence
hypothesis, then Correspondence Analysis will
not deliver any useful information, and can be
ignored.
• Only if the independence hypothesis is rejected
will C.A. be considered as the next step in the
analysis of the pair of variables.
A PCA-like transformation then allows the
modalities of the variables to be represented as
points in factorial planes.
Interpretation
Correspondence analysis plots should be
interpreted by looking at points relative to the
origin
• Points that are in similar directions are
positively associated
• Points that are on opposite sides of the origin
are negatively associated
• Points that are far from the origin exhibit the
strongest associations
www.journalofdentofacialsciences.com
23
Also the results reflect relative associations, not
just which rows are highest or lowest overall
Regression Analysis1: “Refers to any techniques
for modelling and analyzing several variables,
when the focus is on the relationship between
a dependent variable and one or more
independent variables.” Helps to understand how
the typical value of the dependent variable
changes when any one of the independent
variables is varied, while the other independent
variables are held fixed.
TYPES2
¾ Multiple Linear Regression Analysis
¾ Partial Least Square Regression (PLSR)
¾ Principal Component Regression (PCR)
¾ Ridge Regression (RR)
¾ Reduced Rank Regression (RRR) Or
Redundancy Analysis
¾ Poisson’s Regression Analysis
¾ Logistic Regression Analysis
Multiple Linear Regression Analysis (MLR):1
In Multiple linear regressions, several independent
variables are used to predict with a least square
approach one direct variable. If the independent
variables are orthogonal, the problem reduces to a
set of univariate regressions. When the
independent variables are correlated, their
importance is estimated from the partial coefficient
of correlation. An important problem arises when
one of the independent variables can be predicted
from the other variables. This is called
multicolinearity.
The main approaches are:
¾ Forward selection: which involves starting with
no variables in the model, trying out the
variables one by one and including them if
they are ‘statistically significant’?
¾ Backward elimination: which involves starting
with all candidate variables and testing them
one by one for statistical significance, deleting
any that are not significant.
¾ Methods that are a combination of the above,
testing at each stage for variables to be
included or excluded.
Vol. 2 Issue 3
24
Partial Least Square Regression (PLSR)1: It
addresses the multicolinearity problem by
computing latent vectors which explains both the
Independent variables and the Dependent
variables. It is used when the goal is to predict
more than one dependent variables. It combines
features from Principal Component analysis and
Multiple Linear Regression. The score of the units
as well as the loadings of the variables can be
plotted as in Principal component analysis, and the
Dependent variables can be estimated as in
Multiple Linear Regression. It is used to find the
fundamental relations between two matrices
(X and Y). The matrices X and Y are decomposed
into latent structures in an iterative process.1
Uses
¾ Chemometrics
¾ Bioinformatics
¾ Sensometrics
¾ Neuroscience
¾ Anthropology
Principal Component Regression (PCR)1
In Principal Component Regression, the
independent Variables are first submitted to a
Principal Component Analysis and the scores
of the units are then used as predictors in a
standard Multiple Linear Regression.
Step 1= To run Principal component analysis
so as to reduce dimensionality of data.
Step 2= To run a ordinary least squares
regression on selected components; factors
which are most correlated with dependent
variables are selected.
Step 3= Finally the parameters of the model
are computed for the selected explanatory
variables.
Ridge Regression (RR): Ridge Regression
accommodates the multi-colinearity problem by
adding a small constant (the ridge) to the diagonal
of the correlation matrix. This makes the
computation of the estimates for Multiple Linear
Regression possible.
Reduced Rank Regression (RRR) or
Redundancy Analysis4
The Dependent Variables are first submitted to
a PCA and the scores of the units are then used as
www.journalofdentofacialsciences.com
Kumar et al.
Dependent Variables in a series of standard
Multiple Linear Regression where the original
Independent Variables are used as predictors. The
reduced rank regression model is a multivariate
regression model with a coefficient matrix with
reduced rank. It is related to canonical correlations
and involves calculating eigen values and eigen
vectors. It is a non symmetric method .The
components extracted from X are such that they
are closely correlated with the variables of Y as
possible. Similarly, the components of Y are
extracted so that they are closely correlated with
the components extracted from X as possible.
Poissons Regression:2 “form of regression
analysis used to model count data and
contingency tables”. Poisson regression assumes
the response variable Y has a Poisson distribution
and assumes the logarithm of its expected value
can be modelled by a linear combination of
unknown parameters. A Poisson regression model
is sometimes known as a log-linear model,
especially when used to model contingency tables.
Example: Poisson regression is appropriate when
the dependent variable is a count, for instance
of events such as the arrival of a telephone call at a
call centre. The events must be independent in the
sense that the arrival of one call will not make
another more or less likely, but the probability per
unit time of events is understood to be related to
covariates such as time of day.
Logistic Regression:1 Logistic regression
(sometimes called the logistic model or logit
model) is used for prediction of the probability of
occurrence of an event by fitting data to a logit
function logistic curve. It is a generalized linear
model used for binomial regression. Like many
forms of regression analysis, it makes use of
several predictor variables that may be either
numerical or categorical.
Example: The probability that a person has a
heart attack within a specified time period might
be predicted from knowledge of the person's age,
sex and body mass index.
Uses: Logistic regression is used extensively in the
medical and social sciences fields, as well as
marketing applications such as prediction of a
customer's propensity to purchase a product or
cease a subscription.
Vol. 2 Issue 3
Kumar et al.
Multivariate analysis of variance
(MANOVA):7 It is a generalized form of
univariate analysis of variance (ANOVA). It is used
when there are two or more dependent variables.
It helps to answer:
9 Do changes in the independent variable have
significant effects on the dependent variables.
9 What are interactions among the dependent
variables and among the independent
variables.
In MANOVA the Independent Variables have
the same structure as in a standard ANOVA, and
are used to predict a set of Dependent Variables.
MANOVA computes a series of ordered
orthogonal linear combinations of the Dependent
Variables (i.e., factors) with the constraint that the
first factor generates the largest “F” if used in an
ANOVA. The sampling distribution of this F is
adjusted to take into account its construction.
Structural Equation Modelling:8 Structural
Equation Modelling (SEM) is a statistical technique
for testing and estimating causal relations using a
combination of statistical data and qualitative
causal assumptions. Finds useful application in
measurement error, missing data, mediation
model, group differences.
Advanced uses
¾ Invariance
¾ Multiple group comparison
¾ Relations to other types of advanced models
(hierarchical/multilevel models; item response
theory models)
¾ Alternative estimation and testing techniques
Other Methods9,10
Statis: Statis is used when at least one dimension
of the three-way table is common to all tables. The
first step of the method performs a PCA of each
table and generates a similarity table between the
units for each table.
Procrustean Analysis (PA): “Form of statistical
shape analysis used to analyse the distribution of a
set of shapes. To compare the shape of two or
more objects, the objects must be first optimally
"superimposed". This is performed by
optimally translating, rotating and uniformly
scaling the objects. The aim is to obtain a similar
www.journalofdentofacialsciences.com
25
placement and size, by minimizing a measure of
shape difference called the Procrustes distance
between the objects.
CHAID: “Type of decision tree technique, based
upon adjusted significance testing” (Bonferroni
testing). In practice, CHAID is often used in direct
marketing (to select groups of consumers and
predict how their responses to some variables
affect other variables.) medical and psychiatric
research.
Advantages: Output is highly visual and easy to
interpret.
Disadvantages: Requires large sample sizes to
work effectively (Reliable analysis).
CHAID is often used as an exploratory
technique and is an alternative to multiple linear
regression and logistic regression, especially when
the data set is not well-suited to regression
analysis.
Advantages of Multivariate Analyses1
¾ It assures that the results are not biased and
influenced by other factors that are not
accounted for.
¾ Close resemblance to how the researcher
thinks.
¾ Easy visualisation and interpretation of data.
¾ More information is analysed simultaneously,
giving greater power.
¾ Relationship between variables is understood
better.
Application of multivariate analysis1
9 For developing taxonomies or systems of
classification
9 To investigate useful ways to conceptualize or
group items
9 To generate hypotheses
9 To test hypotheses
Finds application in biology, medicine,
psychology, neuroscience, market research,
educational research, climatology, petroleum
geology, crime analysis etc.
Conclusion
There are a variety of multivariate techniques
all of which are based on assumptions about the
nature of the data and the type of association
Vol. 2 Issue 3
 26
under analysis. Choice of an appropriate
procedure to be used in multivariable analysis
depends on whether the dependent and
independent variables are continuous,
dichotomous, nominal, ordinal or a combination
of these. Because there are many potential
problems and pitfalls in the use of multivariable
techniques in clinical research, these procedures
should be used with care.
References
1. Multivariate Analysis; Methods and Applications –
William R Dillon and Matthew Goldstein; Wiley
series 2nd edition
2. Research Methodology- Methods and Techniques-
Kothari CR; New age International. 2nd edition
3. http://www.statisticshell.com/cluster.pdf accessed on
11/05/12
www.journalofdentofacialsciences.com
View publication stats
Kumar et al.
4. Introduction to Biostatistics and Research
Methodology- P.S.S. Sundar Rao, J. Richard, Rao
P. S. S. Sundar, richard J.; 4th edition; PHI
Learning Private.
5. Schaums Outline Of Statistics-Murray R, Spiegel;
2nd edition; Mcgraw-Hill
6. http://www.statsoft.com/textbook/correspondence-
analysis/ accessed on 12/05/2012
7. http://userwww.sfsu.edu/~efc/classes/biol710/mano
va/MANOVAnewest.pdf accessed on 15/05/2012
8. http://www.statsoft.com/textbook/basic-statistics/
accessed on 13/05/2013
9. Methods In Biostatistics: 6th Edition: BK Mahajan;
Jaypee brothers
10. Doing Statistics With SPSS-AW Kerr.;Wiley online
library
Vol. 2 Issue 3