NCM111
MEASUREMENT AND DATA QUALITY
MR NINO ARCHIE S. LABORDO
MEASUREMENT
• Measurement involves rules for assigning numbers to
qualities of objects to designate the quantity of the
attribute. Attributes do not inherently have numeric
values; humans invent rules to measure attributes.
- a statement by early American psychologist L. L.
Thurstone: "Whatever exists, exists in some amount
and can be measured."
• Attributes are not constant; they vary from day to
day, from situation to situation, or from one person
to another.
• Measurement requires numbers to be assigned to
objects according to rules.
- Rules for measuring temperature, weight, and other
physical attributes are familiar to us. Rules for
measuring many variables for nursing studies,
however, have to be created.
Advantages of Measurement
• A major strength of measurement is that it removes
guesswork and ambiguity in gathering and
communicating information
- Consider how handicapped health care professionals
would be in the absence of measures If body
temperature, blood pressure, and so on. Without such
measures, subjective evaluations of clinical outcomes
would have to be used.
• Not all measures are completely objective, but most
incorporate mechanisms for minimizing subjectivity
• Measurement also makes it possible to obtain
reasonably precise information. Precision allows
researchers to make fine distinctions among people
with different degrees of an attribute.
- Instead of describing Nathan as "tall," we can depict
him as being 6 feet 3 inches tall.
• Measurement is a language of communication.
Numbers are less vague than words and can thus
communicate information more clearly.
- If a researcher reported that the average oral
temperature of a sample of patients was "somewhat
high," different readers might develop different
conceptions about the sample's physiologic state. If
the researcher reported an average temperature of
99.6°F, however, there is no ambiguity.
Levels of Measurement
1. Nominal measurement
2. Ordinal Measurement
3. Interval Measurement
4. Ratio Measurement
NOMINAL MEASUREMENT
• The lowest level, involves using numbers simply to
categorize attributes The numbers used in nominal
measurement do not have quantitative meaning
• Examples of variables that are nominallv measured
include gender and blood type.
- If we coded males as 1 and females as 2, the numbers
would not have quantitative implications-the number
2 does not mean "more than" 1.
• Nominal measurement provides information only
about categorical equivalence and nonequivalence
and so the numbers cannot be treated
mathematically.
ORDINAL MEASUREMENT
• Ranks objects based on their relative standing on an
attribute. If a researcher orders people from heaviest
to lightest, this is ordinal measurement.
- As another example, consider this ordinal coding
scheme for measuring ability to perform activities of
daily living:
✓ completely dependent;
✓ needs another person's assistance;
✓ needs mechanical assistance; and
✓ completely independent.
• The numbers signify incremental ability to perform
activities of daily living independently
INTERVAL MEASUREMENT
• Occurs when researchers can specify the ranking of
objects on an attribute and the distance between
those objects. Most educational and psychological
tests yield interval-level measures.
[Date] 1
- For example, the Stanford-Binet Intelligence Scale-a
standardized intelligence (IQ) test used in many
countries-is an interval measure. A score of 140 on the
Stanford-Binet is higher than a score of 120, which, in
turn, is higher than 100. Moreover, the difference
between 140 and 120 is presumed to be equivalent to
the difference between 120 and 100.
RATIO MEASUREMENT
• Ratio measurement is the highest level. Ratio scales,
unlike interval scales, have a rational, meaningful
zero and therefore provide information about the
absolute magnitude of the attribute.
- The Fahrenheit scale for measuring temperature
(interval measurement) has an arbitrary zero point.
Zero on the thermometer does not signify the absence
of heat; it would not be appropriate to say that 60°F
is twice as hot as 30°F.
- Many physical measures, however, are ratio
measures with a real zero. A person's weight, for
example, is a ratio measure. It is acceptable to say
that someone who weighs 200 pounds is twice as
heavy as someone who weighs 100 pounds.
Errors Of Measurement
• Situational contaminants - Scores can be affected by
the conditions under which they are produced. For
example, environmental factors (e.g., temperature,
lighting, time of day) can be sources of measurement
error.
• Response-set biases- Relatively enduring
characteristics of respondents can interfere with
accurate measurements
• Transitory personal factors - Temporary states, such
'as fatigue, hunger, or mood, can influence people's
motivation or ability to cooperate, act naturally, or do
their best.
• Administration variations- Alterations in the
methods of collecting data from one person to the
next can affect obtained scores. For example, if some
physiologic measures are taken before a feeding and
others are taken after a feeding, then measurement
errors can potentially occur.
• Item sampling- Errors can be introduced as a result of
the sampling of items used to measure an attribute.
For example, a student's score on a 100-item test of
research methods will be influenced somewhat by
which 100 questions are included.
Reliability Of Measuring Instruments
RELIABILITY
• Reliability is the consistency with which an
instrument measures the attribute. If a scale
weighed a person at 120 pounds one minute and 150
pounds the next, we would consider it unreliable.
• The less variation an instrument produces in repeated
measurements, the higher its reliability.
• Reliability also concerns a measure's accuracy. An
instrument is reliable to the extent that its measures
reflect true scores-that is, to the extent that
measurement errors are absent from obtained
scores.
• A reliable instrument maximizes the true score
component and minimizes the error component of an
obtained score.
• Three aspects of reliability are of interest to
quantitative researchers: stability, internal
consistency, and equivalence.
STABILITY
• The stability of an instrument is the extent to which
similar results are obtained on two separate
occasions.
• The reliability estimate focuses on the instrument's
susceptibility to extraneous influences over time,
such as participant fatigue.
• Assessments of stability are made through test-retest
reliability procedures. Researchers administer the
same measure to a sample twice and then compare
the scores.
FOR EXAMPLE, THE STABILITY OF A SELF- REPORT SCALE THAT
MEASURED SELF-ESTEEM
[Date] 2
• Because self-esteem is a fairly stable attribute that
does not change much from one day to another, we
would expect a reliable measure of it to yield
consistent scores on two different days.
• As a check on the instrument's stability, we
administer the scale 2 weeks apart to a sample of 10
people
• The scores on the two tests are not identical but, on
the whole, differences are not large.
• Researchers compute a reliability coefficient, a
numeric index that quantifies an instrument's
reliability, to objectively determine how small the
differences are. Reliability coefficients (designated as
r) range from .00 to 1.00.* The higher the value, the
more reliable (stable) is the measuring instrument. In
the example shown in Table 14.1, the reliability
• Test-retest reliability is relatively easy to compute,
but a major problem with this approach is that many
traits do change over time, independently of the
instrument's stability.
• Attitudes, mood, knowledge, and so forth can be
modified by experiences between two
measurements. Thus, stability indexes are most
appropriate for relatively enduring characteristics,
such as temperament.
• Even with such traits, test-retest reliability tends to
decline as the interval between the two
administrations increases
INTERNAL CONSISTENCY
• lnternal consistency reliability is the most widely used
reliability approach among nurse researchers. This
approach is the best means of assessing an especially
important source of measurement error in
psychosocial instruments, the sampling of items.
• Scales and tests that involve summing item scores are
almost always evaluated for their internal
consistency.
• An instrument may be said to be internally consistent
to the extent that its items measure the same trait.
• Internal consistency is usually evaluated by
calculating coefficient alpha (or Cronbach's alpha).
The normal range of values for coefficient alpha is
between .00 and +1.00. The higher the reliability
coefficient, the more accurate (internally consistent)
the measure.
EQUIVALENCE
• Equivalence, in the context of reliability assessment,
primarily concerns the degree to which two or more
independent observers or coders agree about th
scoring on an instrument.
• With a high level of agreement, the assumption is that
measurement errors have been minimized. The
degree of error can be assessed through interrater (or
interobserver) reliability procedures, which involve
having two or more trained observers or coders make
simultaneous, independent observations.
• An index of equivalence or agreement is then
calculated with these data to evaluate the strength of
the relationship between the ratings. When two
independent observers score some phenomenon
congruently, the scores are likely to be accurate and
reliable
Interpretation Of Reliability Coefficients
• Reliability coefficients are important indicators of an
instruments quality. Unreliable measures reduce
statistical power and hence affect statistical
conclusion validity.
• If data fail to support a hypothesis, one possibility is
that the instruments were unreliable-not necessarily
that the expected relationships do not exist.
Knowledge about an instrument's reliability thus is
critical in interpreting research results, especially if
research hypotheses are not supported.
• Various things affect an instrument's reliability. For
example, reliability is related to sample
heterogeneity. The more homogeneous the sample
(i.e., the more similar the scores), the lower the
reliability coefficient will be.
• Reliability estimates vary according to the procedure
used to obtain them. Estimates of reliability
computed by different procedures are not identical,
and so it is important to consider which aspect of
reliability is most important for the attribute being
measured.
VALIDITY
• The second important criterion for evaluating a
quantitative instrument is its validity. Validity is the
degree to which an instrument measures what it is
supposed to measure.
- Suppose we wanted to assess patients' anxiety by
measuring the circumference of their wrists. We could
obtain highly accurate and precise measurements of
[Date] 3
 wrist circumferences, but such measures would not be
valid indicators of anxiety
• Reliability and validity are not totally independent
qualities of an instrument.
- A measuring device that is unreliable cannot possibly
be valid.
- An instrument cannot validly measure an attribute if
it is erratic and inaccurate.
- An instrument can, however, be reliable without
being valid.
• Thus, the high reliability of an instrument provides no
evidence of its validity; low reliability of a measure is
evidence of low validity.
FACE VALIDITY
• Face validity refers to whether the instrument looks
as though it is measuring the appropriate construct,
especially to people who will be completing the
instrument.
- Johnson and colleagues (2008) developed an
instrument to measure cognitive appraisal of health
among survivors of stroke. One part of the
development process involved assessing the face
validity of the items on the scale. Stroke survivors
were asked a series of open-ended questions
regarding their health appraisal after completing the
scale, and then the themes that emerged were
compared with the content of scale items to assess the
congruence of key constructs.
CONTENT VALIDITY
• Content validity concerns the degree to which an
instrument has an appropriate sample of items for
the construct being measured and adequately
covers the construct domain.
• Content validity is crucial for tests of knowledge,
where the content validity question is: "How
representative are the questions on this test of the
universe of questions on this topic?"
• Content validity is also relevant in measures of
complex psychosocial traits. Researchers designing a
new instrument should begin with a thorough
conceptualization of the construct so the instrument
can capture the full content domain.
• Researchers typically calculate a content validity
index (CVI) that indicates the extent of expert
agreement. We have suggested a CVI value of .90 as
the standard for establishing excellence in a scale's
content validity
- Bu and Wu (2008) developed a scale to measure
nurses' attitudes toward patient advocacy. The
content validity of their 84-item scale was rated by
seven experts (a bioethicist, patient advocacy
researchers, measurement experts). The scale's CV1
was calculated to be .85.
• An instrument's content validity is necessarily based
on judgment. No totally objective methods exist for
ensuring the adequate content coverage of an
instrument, but it is increasingly common to use a
panel of substantive experts to evaluate the content
validity of new instruments.
CRITERION-RELATED VALIDITY
• Criterion-related validity assessments, researchers
seek to establish a relationship between scores on
an instrument and some external criterion. The
instrument, whatever abstract attribute it is
measuring, is said to be valid if its scores correspond
strongly with scores on the criterion.
• After a criterion is established, validity can be
estimated easily.
• Criterion-related validity is helpful in assisting
decision makers by giving them some assurance that
their decisions will be effective, fair, and, in short,
valid.
A. Predictive validity - refers to an instrument's ability
to differentiate between people's performances or
behaviors on a future criterion.
For example: When a school of nursing correlates
students' incoming high school grades with their
subsequent grade-point averages, the predictive validity of
high school grades for nursing school performance is being
evaluated.
B. Concurrent validity - refers to an instrument's ability
to distinguish among people who differ in their
present status on some criterion.
For example, a psychological test to differentiate
between patients in a mental institution who could
and could not be released could be correlated with
current behavioral ratings of health care personnel.
[Date] 4
The difference between predictive and concurrent validity, condition correctly. A measure's sensitivity is its rate
then, is the difference in the timing of obtaining measurements of yielding "true positives."
on a criterion. • Specificity. is the measure's ability to identify
noncases correctly, that is, to screen out those
CONSTRUCT VALIDITY
without the condition.
• Construct Validity a key criterion for assessing the • Specificity is an instrument's rate of yielding "true
quality of a study, and construct validity has most negatives."
often been linked to measurement issues. The key
To determine an instrument's sensitivity, and specificity,
construct validity questions with regard to
researchers need a reliable and valid criterion of "caseness”
measurement are:
against which scores on the instrument can be assessed
- "What is this instrument really measuring?" and
"Does it validly measure the abstract concept of
interest?

• The more abstract the concept, the more difficult it is

to establish construct validity however; at the same
time, the more abstract the concept, the less suitable
it is to rely on criterion-related validity. What
objective criterion is there for such concepts as
empathy, role conflict, or separation anxiety?

KNOWN GROUPS

• One approach to construct validation is the known-
groups technique. In this procedure, groups that are
expected to differ on the target attribute are
administered the instrument, and group scores are
compared.
- For instance, in validating a measure of fear of the
labor experience, the scores of primiparas and
multiparas could be contrasted. Women who had
never given birth would likely experience more anxiety
than women who had already had children; one might
question the validity of the instrument if such
differences did not emerge.
- EXAMPLE: suppose we wanted to evaluate whether
adolescents' self-reports about their smoking were
accurate, and we asked 100 teenagers aged 13 to 15
about whether they had smoked a cigarette in the
previous 24 hours.
- The "gold standard" for nicotine consumption is
cotinine levels in a body fluid, and so let us assume
that we did a urinary cotinine assay

FACTOR ANALYSIS

• Factor analysis, which is a method for identifying

clusters of related items on a scale.
• The procedure is used to identify and group together
different measures of some underlying attribute and
to distinguish them from measures of different
attributes
SENSITIVITY, SPECIFICITY AND LIKELIHOOD RATIOS
Calculating Sensitivity, Specificity and Likelihood Ratios
TERMS:
- Sensitivity, in this example, is calculated as the
proportion of teenagers who said they smoked and
who had high concentrations of cotinine, divided by
all real smokers as indicated by the urine test. Put
another way, it is the true-positive findings divided by
all real-positive findings.

• Sensitivity. is the ability of a measure to identify a

- In this case, there was considerable underreporting of
"case" correctly, that is, to screen in or diagnosis a
smoking and so the sensitivity of the self-report was

[Date] 5
 only .50. Specificity is the proportion of teenagers
who accurately reported they did not smoke, or the
true- negative findings divided by all real-negative
findings. In our example, specificity is .83. There was
considerably less over-reporting of smoking ("faking
bad") than under-reporting ("faking good").
(Sensitivity and specificity are sometimes reported as
percentages rather than proportions, simply by
multiplying the proportions by 100.)
8. If a diagnostic or screening tool was used, is
information provided about its sensitivity and
specificity, and were these qualities adequate?
9. Were the research hypotheses supported? If not,
might data quality play a role in the failure to confirm
the hypotheses?

CRITIQUING DATA QUALITY IN QUANTITATIVE STUDIES

Guidelines For Critiquing Data Quality In Quantitative Studies

1. Is there congruence between the research variables

as conceptualized (i.e., as discussed in the
introduction of the report) and as operationalized
(i.e., as described in the method section)?
2. If operational definitions (or scoring procedures) are
specified, do they clearly indicate the rules of
measurement? Do the rules seem sensible? Were
data collected in such a way that measurement errors
were minimized?
3. Does the report offer evidence of the reliability of
measures? Does the evidence come from the
research sample itself, or is it based on other studies?
If the latter, is it reasonable to conclude that data
quality would be similar for the research sample as for
the reliability sample (e.g., are sample characteristics
similar)?
4. If reliability is reported, which estimation method was
used? Was this method appropriate? Should an
alternative or additional method of reliability
appraisal have been used? Is the reliability sufficiently
high?
5. Does the report offer evidence of the validity of the
measures? Does the evidence come from the
research sample itself, or is it based on other studies?
If the latter, is it reasonable to believe that data
quality would be similar for the research sample as for
the validity sample (e.g., are the sample
characteristics similar)?
6. If validity information is reported, which yalidity
approach was used? Was this method appropriate?
Does the validity of the instrument appear to be
adequate?
7. If there is no reliability or validity information, what
conclusion can you reach about the quality of the data
in the study?

[Date] 6