Reliability ,Validity & the Problems of

Psychiatric Diagnoses

Presented by :Mr. Sarin Dominic

Chairperson : Ms. Sahely Ganguly

1
 Outline of the presentation
Psychiatric classification and diagnosis – concept,
dimensions, purpose and limitations
History of psychiatric classification
Current classificatory systems
Reliability of psychiatric diagnosis
Validity of psychiatric diagnosis
Research findings

2
 Introduction
 Diagnosis is regarded as a sine qua non for clinical practice
and research. It provides information about patients
symptom profiles, prognosis, treatment outcomes and sets
the boundaries for research through delineating
homogeneous patient groups.
 Unlike other branches in medicine where there is better
understanding of the underlying biological processes, in
psychiatry the diagnoses are still based on identification of
clinical syndromes.

3
  The introduction of explicit operational criteria and rule-
based classifications significantly improved diagnostic
agreement.
 But still we have not crossed the limitations. There are lot of
issues with diagnoses based on current systems
 Owing to the lack of objective measurements for making
definitive diagnoses, operationalised diagnoses should
therefore be regarded as provisional and its reliability and
validity to be established.
(Maria, M. et al., 2007)

4
 Key Terms
Diagnosis
The word stems from dia (Greek) meaning through and
gnosis (Greek) meaning knowledge, or the establishing of
the nature of a disease.
It is defined the use of specific taxonomic schemes or
classification systems to identify illness.
Disorder
A term midway between a disease or illness and a
syndrome, in terms of consistency, correlates and
significance (Jablensky & Kendell, 2002)

5
 Reliability
The extent to which an experiment, test or any measuring
procedure yields the same results on repeated trials
(Carmines & Zeller, 1979)

Validity
The extent to which an empirical indicator of the concept
actually represents the concept of interest (Anastasi,
1976)

6
 Purpose of Classification and Diagnosis
 Communication
 Standard frame of reference
 Common language
 Meet administrative and legal needs
 Control
 Determining Treatment options
 Prevention and control
 Comprehension
 Organize and simplify clinical history
 Scientific study
 Exploring etiology
 Predict clinical course
(Dalal & Sivakumar, 2009)
7
 Disadvantages
 Labeling – Stigma associated with ‘labels’
(Wright, A. et al., 2011)
 Categories based on signs and symptoms are not perfect
(when etiology is ignored)

8
 Nature of psychiatric diagnosis and
classification
 Unit of classification – Disorder
 Disorders are arranged into discrete categories
 Polythetic
 Classification can be based on clinical syndrome,
pathological process, by deviation from norm, by
hypothetical process and by etiology

9
  Kendell described that every patient possesses three kinds
of characteristics:
A. Universal, shared with all others
B. Shared with some but not others
C. Unique, shared with no others

 Where A dominates, classification is pointless; where C

dominates, classification is impossible. The presence of B is
the crucial one and so classification depends on B relative
to A and C.

10
 Dimensions of Categorization
 Causalism Vs. Descriptivism
 Essentialism Vs. Nominalism
 Objectivism Vs. Evaluativism
 Internalism Vs. Externalism
 Categories Vs. Continua
 Entities Vs. Agents
(Zachar & Kendler, 2007)

11
 Causalism Versus Descriptivism
 Should psychiatric disorders be categorized as a function of
their causes (causalism) or their clinical
characteristics(descriptivism) ?
 Three different approaches can be taken to the role of
causalism in psychiatric nosology.
 Temporizing
 Robust descriptivism
 Intermediate position

12
 Essentialism Vs. Nominalism
 Are categories of psychiatric disorder defined by their
underlying nature (essentialism), or are they practical
categories identified by humans for particular uses
(nominalism) ?
 There are two approaches to nominalism.
 Radical nominalism
 Moderate nominalism

13
 Objectivism Vs. Evaluativism

 Is deciding whether or not something is a psychiatric

disorder a simple factual matter (objectivism), or does it
inevitably involve a value-laden judgment (evaluativism)?

14
 Internalism Versus Externalism
 Should psychiatric disorders be defined solely by processes
that occur inside the body (internalism), or can events
outside the skin also play an important (or exclusive)
defining role (externalism)?
 Externalists come in two sorts.
 Radical externalists
 Moderate externalists

15
 Categories Versus Continua
 Are psychiatric disorders best understood as illnesses with
discrete boundaries (categorical) or the pathological ends
of functional dimensions (continuous)?

16
 Entities Vs. Agents
 Should psychiatric disorders be considered to be “things”
people get, or are they inseparable from an individual’s
personal subjective makeup?

17
 History of Psychiatric classification
Many systems of Classification has been developed in the last
millennia have differed in their relative emphasis on
phenomenology, etiology, and course as defining features of
psychiatric illness.

18
 Ancient era
Hippocrates : Imbalance of four Humours
18th Century
Scientific concepts of psychopathology evolved
Pathology were grouped into syndromes
Botanical Taxonomic approach
Boissier de Sauvages
William cullen
Philip Pinel
Esquirol

19
 19th Century
Terms Psychosis and neurosis were coined
Anatomical-clinical approach
Karl Kahlbaum and Ewald Hecker : Descriptive
categorization of syndromes
Wilhelm Griesinger : Brain pathology
Thomas Coulston : developmental insanity
Emil Kraepelin : proposed 15 categories

20
 20th Century
Brain disease (or degeneration) model
Psychoanalytic model: neurosis, psychosis, and
perversion
Karl Jaspers : Phenomenology and biographical method
Adolph Meyer : mixed biosocial scheme
William C. Menninger : Medical 203
1908 : Statistical Manual for the use of institutions for
the Insane was issued
1935 : Standard classified nomenclature of disease was
issued

21
 Development of international Classificatory
systems

 1949 : Mental disorders were first included in the sixth revision

of the International Classification of Diseases (ICD-6) - less
successful
 1952 : APA created own classification system, DSM-I
 1955 : ICD 7 was published – no changes
 1966 : ICD 8 was published
 1968 : DSM II was published
expert opinion
psychoanalytic concepts of etiology
poor inter-rater reliability
 1972 : Feighner criteria was published

22
 1975 : ICD 9 was published
health statistics
1980 : DSM-III was published
explicit diagnostic criteria
multiaxial system
descriptive approach
1992: ICD 10 was published
Several different descriptions
1994 : DSM IV was published
Systematic reviews
Reanalyses of data sets
Field trials
2013 : DSM V was published

23
 Current classification Systems
 International Statistical Classification of Diseases and
Related Health Problems 10th Revision Chapter V- Mental
and Behavioural Disorders - WHO
 Diagnostic and Statistical Manual of Mental Disorders, Fifth
Edition. - APA

24
 Issues with current classificatory systems
 Definition of mental disorder and threshold for diagnosis
 Multiple diagnoses
 Model of classification (Categorical vs. Dimensional
 Culture and values
 Gender
 Inclusion of laboratory tests within criterion sets
(Widiger, 2005)

25
 Problems of Psychiatric Diagnoses
 Relies on the patient’s own subjective report of symptoms
and the doctor’s observation of patient behavior to arrive
at a diagnosis. (Hyman, 2003)
 Lacks objective and independent criteria for sorting out
psychiatric disorders. (Hyman, 2003)
 Manifested by a quantitative deviation in behavior,
ideation and emotion from a normative concept and it is
difficult to define normal human behavior. (Zimmerman &
Spitzer, 2005)

26
  Highly nonspecific and quite unstable over time. (Benzato,
2004)
 The task of diagnosis is far more complex in Psychiatry
than in general medicine because “even the typical form of
a given mental disorder is often obscured by illness
manifestation shaped by personality and sociocultural
circumstances (Gupta A., 1991)

27
 Reliability of psychiatric diagnosis
 Before the 1970s, psychiatric research and communication
among clinicians were badly hampered by the low
reliability of diagnostic assessment .(Jablensky & Kendell,
2003 )
 The key terms like schizophrenia were used in different
ways in different countries, or even in different centres
within a single country. (Stengel,1959)

28
  The situation has changed radically since then, and
particularly since the publication of DSM-III in 1980 and
the research version of ICD-10 in 1993. Clearly, this has
been largely the result of the introduction of explicit or
operational diagnostic criteria. (Jablensky & Kendell, 2003 )
 Reliability of diagnosis can only be guaranteed to the
extent that a classificatory system (ICD 10, DSM 5, CCMD
3) is reliable and is used by different psychiatrists and the
same psychiatrist at different times, to make the same
diagnosis.

29
 Types of reliability
Inter-rater
 It is a measure of how often two or more people arrive at
the same diagnosis given an identical set of data.
(Matuszak et al. , 2012)
Test-retest
 The degree to which the initial diagnosis is retained by the
same clinician over period of time.
 Initial diagnosis of schizophrenia, especially in women, was
often changed as the psychiatrists got to know their
patients better. The extent of the reliability of the first
diagnosis is therefore limited. (Seeman, 2007)

30
 Causes of unreliability of psychiatric diagnosis
Ward, et al.(1962), conducted a study to pinpoint the
reasons for diagnostic disagreement among
psychiatrists.
The three main reasons for diagnostic disagreement
were
Inconstancy of the patient (5%),
Inconstancy of the clinician (32.5%)
Inadequacy of the nomenclature (62.5%)

31
 Patient factors
 Patient's psychological state
 The use of proxy information
 Atypical presentations of psychiatric disorders
(Aboraya, et al., 2006)

32
 Clinician factors
 The clinician interview
 Clinician training, experience, and school of thoughts
 Reliance on observation
 Psychiatric nomenclature
(Aboraya, et al., 2006)

33
 Ways to improve reliability
Structuring and standardizing the assessment procedure
Improving the training of both respondents and those
carrying out the assessment
Averaging replicating measurements
(Shrout , 2011)

34
 Validity
 The word “valid” is derived from the Latin validus, meaning
strong
 It is defined as “well founded and applicable; sound and to
the point; against which no objection can fairly be
brought” (Shorter Oxford English Dictionary, 3rd ed.,
1978)
 What is meant by the validity of a diagnostic concept, or of
a system of classification in psychiatry, is rarely discussed
and few studies have addressed this question explicitly and
directly. (Jablensky & Kendell, 2002)

35
 Types of validity
Four types of validity are often mentioned in the
discourse on psychiatric diagnosis
Construct validity
 Whether diagnostic category is based on a coherent,
explicit set of defining features ?
Content validity
 Whether it has empirical referents, such as verifiable
observations for establishing its presence ?

36
 Concurrent validity
Whether it can be corroborated by independent
procedures such as biological or psychological tests?
Predictive validity
Whether it predicts future course of illness or treatment
response?
(Jablensky & Kendell, 2002)

37
 Validity Criterion
‘Any knowledge, method or procedure that can improve the
accuracy of the disease, syndrome, or disorder
measurement, help to rule out other diseases, syndromes, or
disorders in the differential diagnosis, or validate a
provisional diagnosis of the disease, syndrome, or disorder’-
(Aboraya, 2004)

38
 Models to increase validity
In 1970, Robins and Guze, proposed five phases to achieve
valid classification of mental disorders They are
Clinical description
Laboratory study
Exclusion of other disorders
Follow-up study
Family study

39
 This schema was elaborated by Kendler (1980) who
distinguished between
Antecedent validators (familial aggregation, premorbid
personality, and precipitating factors)
Concurrent validators (including psychological tests)
Predictive validators (diagnostic consistency over time,
Rates of relapse and recovery, and response to
treatment)

40
 Andreasen (1995) proposed “a second structural
program for validating psychiatric diagnosis” and listed
several additional validators
Molecular genetics
Molecular biology
Neurochemistry
Neuroanatomy
Neurophysiology
Cognitive neuroscience study

41
 Research on additional validators (endophenotypic
markers)
 Many impulse control disorder are associated with low
level of serotonin metabolites. (Moellar et al., 2001)
 Major depression is frequently associated with left frontal
hypoactivation (Henriques & Davidson, 1991)
 Increases in urinary cortisol production, levels of
corticotropin-releasing hormone (CRH) in spinal fluid
(Nemeroff, 1996), and a general disturbance in the normal
pattern of cortisol secretion have been identified in
depression.(Carroll et al., 1981).

42
  In studies of monozygotic twins discordant for
schizophrenia, the affected twin usually has bigger
ventricles (Stabenau &Pollin, 1993)
 PTSD patients show evidence of an enhanced negative
feedback inhibition characterized by an exaggerated
cortisol response to dexamethasone, an increased number
of glucocorticoid receptors, and lower basal cortisol levels
(Yehuda, 1998)

43
 Obstacles in using endophenotypic markers in
psychiatric diagnosis
Conceptual : There is widespread assumption that
endophenotypic markers are closely linked underlying
etiological processes than exophenotypic markers
( Kihlstorm, 2002)
 Eg: Diminished amplitude of P300 in ADHD

Empirical : No endophenotypic marker yet identified are

close to serving as inclusion tests for the respective
disorders (Keri & Janka, 2004)
 Eg. Smooth pursuit eye movement in schizophrenia

44
 Kendell and Jablensky (2003):
One of two conditions to be met
 Zones of rarity between one syndrome and others.
Clear qualitative differences in biological underpinnings
of one syndrome and another.
In the absence of clinicopathological correlates, it is
difficult to verify psychiatric syndromes.

45
 Spitzer (1983) proposed the LEAD standard.
 LEAD is an acronym for longitudinal expert all data
 The LEAD standard is an important step toward obtaining
the best estimate diagnosis by requiring expert clinicians
to utilize all the available data over time, including
information from family members, hospital records,
psychological evaluation, and laboratory results.

46
  The requirement of LEAD to have expert clinicians make
independent assessments, discuss diagnostic
disagreement, and make a consensus diagnosis accounts
for the difficulty in implementing the LEAD standard and its
limited use.

47
 Measuring validity
The common measure used in research to assess
validity of psychiatric diagnosis are
Statistical measures of prospective and retrospective
consistency of diagnosis
External validators (endophenotypic markers) like brain
imaging techniques, CSF analysis, blood profiling

48
 Prospective consistency
The proportion of subjects in a category at baseline
assessment who retained the same diagnosis at the end of
the follow-up period.
Retrospective consistency
The proportion of subjects whose diagnosis at the end of
follow-up is the same as that made at the baseline
assessment.
(Schwartz et al., 2000)

49
 Relating reliability and validity
 Reliability of a diagnostic classification tells us little about
the validity of its rubrics. In fact, a highly reliable diagnostic
system can be of dubious validity. (Jablensky & Kendell,
2003 )
 On the other hand, a diagnostic concept of demonstrable
validity one with important external correlates like
neurocognitive features, familial aggregation of cases, or
prediction of treatment response, may command poor
diagnostic agreement.

50
  Reliability imposes a ceiling on the evaluation of validity in
the sense that validity would be extremely difficult to
determine if the diagnostic category was unreliable. (Rice
et. al., 1992)

51
 Inter rater reliability studies
A study was conducted which assessed the extent of
agreement on psychiatric diagnosis in written evidence
provided by experts in serious criminal matters in Australia.
Good inter-rater agreement was found on the diagnoses of
acquired brain injury, schizophrenia-spectrum psychosis,
substance-induced psychotic disorder, and intellectual
disability.
Moderate agreement was seen on the diagnosis of depressive
and personality disorders.
Agreement on anxiety disorders, in particular post-traumatic
stress disorder, was poor.
(Nielssen et al.,2010)

52
 Reliability of diagnosis using structured and
unstructured measures
 Another study measured interrater reliability for diagnoses
between three groups as achieved by Traditional
Diagnostic Interview (TDI) vs. Computer Assisted
Diagnostic Interview (CADI).
 Group 1 (33) - TDA, 45.5%
 Group 2 (39) - CADI, 79.5%
 Group 3 (33) - TDA, 54.5%
 The results demonstrated that CADI had better interrater
reliability than TDA.

53
 Comparative reliability
A study was conducted to assess comparative reliability
of ICD-9, ICD-10 and DSM-III-R.
Random sample of 100 new patients referred
consecutively to the psychiatric hospital was assessed.
Overall reliability and reliability of each major
psychiatric diagnosis were compared between the three
systems.
ICD-10 was found to have the highest reliability figures.
(Okasha et al., 1993)

54
 Test – Retest reliability (DSM V Field Trial)
 Regier et al.,( 2013) conducted DSM-5 Field Trials in the
United States and Canada to find out Test-Retest Reliability
of Selected Categorical Diagnoses
 Estimates of the intraclass kappa as a measure of the
degree to which two clinicians could independently agree
on the presence or absence of selected DSM-5 diagnoses
when the same patient was interviewed on separate
occasions, in clinical settings, and evaluated with usual
clinical interview methods was obtained.

55
  There were a total of 15 adult and eight child/adolescent
diagnoses for which adequate sample sizes were obtained
to report adequately precise estimates of the intraclass
kappa.
 Very good range (kappa=0.60–0.79) –
PTSD,
Complex somatic symptom disorder
Major neurocognitive disorder
ADHD
Autism Spectrum Disorders

56
  Good range (kappa=0.40–0.59) :
 Schizophrenia
 Schizoaffective disorder
 Bipolar I disorder
 Binge eating disorder
 Alcohol use disorder
 Mild neurocognitive disorder
 Borderline personality disorder
 Oppositional defiant disorder
 Avoidant/restrictive food intake disorder

57
  Questionable range (kappa=0.20–0.39) :
 Major depressive disorder,
 Generalized anxiety disorder
 Mild traumatic brain injury
 Antisocial personality disorder
 Disruptive mood dysregulation disorder

58
  Unacceptable range (kappa values,0.20)
 Mixed anxiety and depressive disorder
 Nonsuicidal self-injury disorder
 Eight other diagnoses had insufficient sample sizes to
generate precise kappa estimates at any site.

59
 Stability of Psychiatric Diagnoses
 Tewfik K. Daradkeh (UAE, 2006) studied the diagnostic
stability of psychiatric diagnoses over a 4-year period using
percentages of index diagnoses that did not change over
time.
o Highest diagnostic stability was found in patients with
index diagnoses of alcohol abuse, schizophrenia and
drug abuse.
o Lowest stability was found in patients with depressive
disorders, acute psychoses and bipolar disorders

60
 Pope, et al. (2013) conducted a study to confirm the
diagnostic stability of FEP disorders over 1 year.
Psychotic disorder diagnoses were retained by 76.2% of
patients at 1 year, schizophrenia being the most stable
diagnosis (92.1%).
Comorbid SUDs, anxiety disorders, and mood disorders
persisted for 50.7%, 64.0%, and 16.7% of patients,
respectively.

61
 Chung, et al. (2009) conducted a systematic review of
articles concerned with evaluating the diagnostic
stability of functional psychosis.
Schizophrenia was found to be the most stable diagnosis
followed by affective psychosis. Other psychotic
disorders were diagnostically unstable over time. Around
one-fifth of patients with first-onset psychosis had their
diagnoses revised at follow-up.

62
 Paul, et al. (1991) examined the reliability, stability, and
clinical and predictive validity of Axis II diagnosis in 40
patients with major depression.
Symptoms were assessed with the Personality Disorder
Examination (PDE) and the Personality Assessment Form
(PAF)
Concordance among the 3 methods was limited, with
unanimous agreement on 20 of 40 cases (50%) at intake
and 18 of 31 cases (58%) at follow-up.
Patients with personality disorders identified by the LEAD
standard consensus had worse outcomes at the 6-mo
follow-up assessment.

63
 Garcia, et al., (2007) conducted a study in a Spanish
hospital to evaluate the long-term stability of the
most prevalent psychiatric diagnoses in a variety of
clinical settings.
10025 adult patients who were assessed on at least ten
in three settings: in-patient unit, emergency room, and
out-patient psychiatric facilities.
Prospective consistency, retrospective consistency and
the proportion of patients who received each diagnosis
in at least 75% of the evaluations were calculated for
each diagnosis in each setting and across settings.

64
65
66
 Conclusion
 Establishing reliability and validity of diagnosis was a tough
task and will be too in psychiatric diagnosis until the
etiology is known.
 Efforts to make diagnosis reliable by way of creating
operational criteria and establishing international
classificatory systems have given rise to the concept of
diagnostic entities. But these entities could become real
only when we are able to define the real causatives.

67
  Our classificatory systems will continue to evolve with
advances in science especially in neuroscience,
neurobiology and genetics which gives hope that we would
be able to tackle these issues in the future.

68
 Reference
Aboraya, A. et al., (2005). The Validity of Psychiatric Diagnosis Revisited
The Clinician's Guide to Improve the Validity of Psychiatric Diagnosis,
Psychiatry (Edgmont). 2(9): 48–55.
American Psychiatric Publishing, (1994). Diagnostic and Statistical
Manual of Mental Disorders, Fourth Edition. Washington, DC:
Andreasen, N. C., (1995). The validation of psychiatric diagnosis: New
models and approaches. American Journal of Psychiatry, 152(2), 161–2.
Banzato, C. E. (2008). Critical evaluation of current diagnostic systems.
Indian Journal of Psychiatry, 50,155-7.
Banzato, C.E. (2004). Classification in psychiatry: The move towards
ICD-11 and DSM-V. Current Opinion Psychiatry, 17, 497-591.

69
 Bessel, A. (2004). Psychobiology of Post Traumatic Stress Disorder, In
Panksepp J. (Ed.).Textbook of Biological Psychiatry, pp. 324-325, New
Jersey: Wiley-Liss, Inc.
Chang, W. C., (2009) Diagnostic Stability of Functional Psychosis: a
Systematic Review, Hong Kong Journal of Psychiatry, 19, 30-41
Dalal, P.K. & Sivakumar, T. (2009). Moving towards ICD-11 and DSM-V:
Concept and evolution of psychiatric classification. Indian Journal of
Psychiatry, 51, 310-19.
Feighner, J.P. et al. (1972) Diagnostic criteria for use in psychiatric
research. Arch Gen Psychiatry, 26, 57–63.
Goldstein J. et al. (2002). Validity. Definitions and Applications to
Psychiatric Research. In Tsuang, M.T. & Tohen, M. Textbook in Psychiatric
Epidemiology. pp 99-113, New York: Wiley-Liss.

70
 Jablensky, A., Kendell, R.E. (2002). Criteria for assessing a classification in
psychiatry. In: Maj, M., Gaebel, W., et al., editors. Psychiatric diagnosis
and classification. Wiley. pp. 1-24. West Sussex:
Lilenfield, S. et al., (2013). Issues in diagnosis, conceptual issues and
controversies. In Edward W. et al.(Eds.), Psychopathology History,
Diagnosis, and Empirical Foundations pp. 1-28, John-Wiley and sons,
New Jersey retrieved from www.books.google.com as on June 2, 2014.
Maria M. et. al. (2007). Diagnostic stability of psychiatric disorders in
clinical practice British Journal of Psychiatry, 190, 210-216.
Mayberg, H.S. (2004). Depression, A Neuropsychiatric Perspective, In
Panksepp J. (Ed.).Textbook of Biological Psychiatry, pp. 203-207, New
Jersey: Wiley-Liss, Inc.

71
 Natalie D et al. (2004). Diagnostic stability in a Dutch psychosis
incidence cohort, British Journal of Psychiatry 185, 460-464.
Nielssen, O. (2010). The Reliability of Evidence About Psychiatric
Diagnosis After Serious Crime: Part I., Journal of American Academy of
Psychiatry and the Law, 38(4), 516-523 retrieved from
www.ncbi.nlm.nih.gov/pubmed on June 3, 2014
Okasha, A et al. (1993). Diagnostic agreement in psychiatry. A
comparative study between ICD-9, ICD-10 and DSM-III-R. The British
Journal of Psychiatry 162, 621-626.
Pilkonis, A. et al. (1991). Validity in the diagnosis of personality
disorders: The use of the LEAD standard. By Psychological Assessment:
Journal of Consulting and Clinical Psychology, Vol 3(1), 46-54. Retrieved
from http://psycnet.apa.org/ on May 24, 2014

72
 Robins, E. & Guze, S.B.(1970) Establishment of diagnostic validity in psychiatric
illness: Its application to schizophrenia. American Journal of Psychiatry,
126(7):983–7.
Rosenhan, D.L. (1972). On being sane in insane places. Science, 179, 250-8.
Shrout, P.E. (2002). Reliability. In Tsuang, M.T. & Tohen, M. Textbook in Psychiatric
Epidemiology. pp 73-83, New York: Wiley-Liss.
World Health Organization, (1992). International Classification of Disease
10:Clinical descriptions and diagnostic guidelines. Geneva
Wright, A. et al. (2011). Labelling of mental disorders and stigma in young people.
journal of social science and medicine Aug;73(4), 498-506. doi: 10.1016/
j.socscimed.2011.06.015. retrieved from www.pubmed.gov on May 24, 2014
Zachar, P. & Kendler, K., (2007). Psychiatric Disorders: A Conceptual Taxonomy
American Journal of Psychiatry, 164, 557–565.

73
 THANK YOU

74