5/15/2021 Assessing Questionnaire Validity - Select Statistical Consultants

D
Blog

Meet the Team

News

Case Studies

Blog

Search by Sector

Healthcare (5)
Questionnaire surveys are measurement instruments. While scientific Education (9)
measurement instruments measure physical properties like velocity or Public Sector (6)
weight, questionnaire surveys often measure respondents’ self-reported Business (12)
attitudes, opinions or behaviours. They are used frequently in social Market Research (11)
science to measure what psychologists call constructs. Academic (2)
Charities (1)
 

As constructs are intangible and complex human behaviours or Search by Service

characteristics, they are not well measured by any single question. They
Design (3)
are better measured by asking a series of related questions covering
Modelling (10)
different aspects of the construct of interest. The responses to these
Surveys & Sampling
individual but related questions can then be combined to form a score or
(10)
scale measure along a continuum.
Analysis (14)
Visualisation (11)
Reliability and validity

https://select-statistics.co.uk/blog/assessing-questionnaire-validity/ 1/5
5/15/2021 Assessing Questionnaire Validity - Select Statistical Consultants

Prediction (5)
Reporting (1)
Data Collection &
Management (2)
Testing (5)
Advice (4)
Data Mining (5)
Qualitative Analysis (1)

As with scientific measurement instruments, two important qualities of

surveys are consistency and accuracy. These are assessed by considering

the survey’s reliability and validity.

Following on from our previous blog which looked at approaches to

assessing reliability, this blog focusses on ways to assess a survey’s

validity.

As with reliability, the validity of a survey can be assessed in a number of

different ways and the methods to choose will depend on the survey
design and purpose. Often it is desirable to use more than one to

facilitate a more rounded judgement of validity.

Validity

Validity is the extent to which an instrument, a survey, measures what it is

supposed to measure: validity is an assessment of its accuracy.

How do we assess validity?

Face validity and content validity are two forms of validity that are usually

assessed qualitatively. A survey has face validity if, in the view of the

respondents, the questions measure what they are intended to measure.

A survey has content validity if, in the view of experts (for example, health

professionals for patient surveys), the survey contains questions which

cover all aspects of the construct being measured.

Face and content validity are subjective opinions of non-experts and

experts. Face validity is often seen as the weakest form of validity, and it

is usually desirable to establish that your survey has other forms of

validity in addition to face and content validity.

Criterion validity is the extent to which the measures derived from the

survey relate to other external criteria. These external criteria can either

be concurrent or predictive.

https://select-statistics.co.uk/blog/assessing-questionnaire-validity/ 2/5
5/15/2021 Assessing Questionnaire Validity - Select Statistical Consultants

Concurrent validity criteria are measured at the same time as the survey,
either with questions embedded within the survey, or measures obtained

from other sources. It could be how well the measures derived from the

survey correlate with another established, validated survey which

measures the same construct, or how well a survey measuring affluence

correlates with salary or household income.

Often the purpose of a survey is to make an assessment about a situation

in the future, say the suitability of a candidate for a job or the likelihood

of a student progressing to a higher level of education. Predictive validity

criteria are gathered at some point in time after the survey and, for

example, workplace performance measures or end of year exam scores

are correlated with or regressed on the measures derived from the

survey.

If the external criteria is categorical (for example, how well a survey

measuring political opinion distinguishes between Conservative and

Labour voters), while still criterion validity, how well a survey

distinguishes between different groups of respondents is referred to as

known-group validity. This could be assessed by comparing the average

scores of the different groups of respondents using t-tests or analysis of

variance (ANOVA).

Construct validity is the extent to which the survey measures the

theoretical construct it is intended to measure, and as such encompasses

many, if not all, validity concepts rather than being viewed as a separate

definition.

Confirmatory factor analysis (CFA) is a technique used to assess construct

validity. With CFA we state how we believe the questionnaire items are

correlated by specifying a theoretical model. Our theoretical model may

be based on an earlier exploratory factor analysis (EFA), on previous

research or from our own a priori theory. We calculate the statistical

likelihood that the data from the questionnaire items fit with this model,

thus confirming our theory.

This blog doesn’t provide an introduction to factor analysis, we’ll post an

article on this topic in the future. Here we explain how factor analysis is

used in the context of validity.

Below is a diagram representing a simple theoretical model.

https://select-statistics.co.uk/blog/assessing-questionnaire-validity/ 3/5
5/15/2021 Assessing Questionnaire Validity - Select Statistical Consultants

Here there are five questionnaire items (labelled Q1 to Q5 in the diagram

above), each of which is measured with a component of error or

uncertainty (labelled e1 to e5). The model hypothesises that an individual’s

responses to each of the survey questions is influenced by the underlying

latent construct, the factor. The construct is “latent” because it is not

directly observed, it is observed or measured through responses to

questions related to the construct.  For example, respondents with higher

levels of self confidence will be more likely to endorse statements such as

“I am happy being the centre of attention” or “I feel comfortable talking to

people I don’t know“, whereas respondents with lower levels of self-

confidence are more likely to disagree with these statements. So

respondents’ questionnaire responses are driven by, and correlated with,

their underlying characteristic.

This kind of model is known as a factor analysis model. It shows how the

correlations between the questionnaire items can be explained by

correlations between each questionnaire item and an underlying latent

construct, the factor. These correlations are known as factor loadings and

are represented by arrows between the latent factor and the

questionnaire items.

By fitting the model we can estimate these factor loadings. We then

compare the estimates of the factor loadings with their standard errors

and calculate the likelihood that these are different from zero, and

therefore how much statistical evidence there is to support our

hypothesis (that the theoretical factor analysis model fits the data). We

can also compare the fit of the model overall with goodness of fit

statistics such as the model Chi-squared, the Comparative Fit Index and

the Root Mean Square Error of Approximation.

Using confirmatory factor analysis we test the extent to which the data

from our survey is a good representation of our theoretical understanding

https://select-statistics.co.uk/blog/assessing-questionnaire-validity/ 4/5
5/15/2021 Assessing Questionnaire Validity - Select Statistical Consultants

of the construct; it tests the extent to which the questionnaire survey

measures what it is intended to measure.

Relationship between validity and reliability

While reliability and validity are two different qualities, they are closely

related and interconnected.   A survey can have high reliability but poor

validity. A survey, or any measurement instrument, can accurately

measure the wrong thing. For example, a watch that runs 10 minutes fast.

However, for a survey, or measurement instrument, to have good validity

it must also have high reliability. Without good reliability a survey is not

validly measuring what it is intended to measure: it is measuring

something else (other constructs or noise). Reliability is a necessary but

not sufficient condition for validity.

g Older Newer h

https://select-statistics.co.uk/blog/assessing-questionnaire-validity/ 5/5