Expanding the Framework for
Mental Health Program Evaluation1
Peter D. Fox, PhD, and John M. Kuldau, MD, Stanford, Calif
WHILEHILE there is an extensive literature
on the evaluation of mental health pro-
grams, the state of the art in mental health
program evaluation is unsettled, given the
various competing criteria and methodolo-
gies that have been proposed. In spite of the
long history of psychiatry, psychiatric pro-
grams remain among the most difficult to
evaluate in the field of medicine.
A comprehensive view of a psychiatric
program must encompass three related ac-
tivities that are usually performed by groups
of workers who are separated by training,
tradition, and organizational barriers. These
activities are (1) administrative and finan-
cial, involving the assignment of resources,
eg, mental health workers, to a program; (2)
clinical treatment; and (3) assessment of
treatment outcome. In spite of the strong in-
terrelationships, the three factors are usual-
ly treated as separate by the social system in
which they are imbedded.
This paper offers new approaches to eval-
uating mental health programs by broaden-
ing the framework of evaluation through the
consideration of all three activities. The first
section discusses the need for a more syste¬
matic approach to program evaluation. Next,
some current assessment practices are pre¬
sented, followed by discussion of the rele¬
a tates good decision making by providing the
vance of economic criteria to psychiatric pro-
Submitted for publication Feb 20,1968.
From the Department of Psychiatry, Stanford
University School of Medicine, and the Stanford Uni-
versity Graduate School of Business, Stanford, Calif
(Dr. Fox), and Stanford University School of Medi-
cine and Palo Alto Veterans Administration Hos-
pital (Dr. Kuldau). Dr. Fox is currently at the
Stanford Research Institute, Menlo Park, Calif.
Reprint requests to the Department of Psychi-
atry, Stanford University School of Medicine, Stan-
ford, Calif 94305 (Dr. Kuldau).
Downloaded From: by a Tulane University User on 01/11/2019
gram evaluation. The basic concepts of cost-
effectiveness analysis and its relevance to
program evaluation are then described. Fi¬
nally, some issues and challenges that fre¬
quently arise in evaluating psychiatric
programs are discussed. The paper refers pri¬
marily to psychiatric programs that receive
support from the community or the govern¬
ment, although much of the paper is relevant
to psychiatric programs in general, inde¬
pendent of the sources of funding.
Need for a More Systematic Approach
There are at least three reasons to strive
for a more systematic and rigorous approach
to the evaluation of psychiatric programs.
The first is the necessity of choice. Profes¬
sionals in the mental health field, be they
psychiatrists, social workers, psychologists,
or other personnel, are faced with the prob¬
lem of selecting among various programs or
modes of treatment designed to assist the
patient. This choice is forced because it is
not possible to undertake simultaneously all
forms of treatment and rehabilitation. Yet
there is considerable disagreement regard¬
ing the most desirable approaches to treat¬
ment. Meaningful program evaluation facili¬
information required to compare alternative
programs.
Second, evaluation is an integral part of
good clinical practice. Good clinical practice
demands that the practitioner define the
problems of each individual patient as accu¬
rately as possible, describe the course of ac¬
tion to be pursued, predict the effects of an
intervention, and finally, receive informa¬
tion of the effects of this intervention. For
 example, a surgeon who performs an appen¬ other federal agencies including those dealing
dectomy formally states his diagnosis of ap¬ with health. These accounting systems, re¬
pendicitis in the hospital records, operates, ferred to as planning-programing-budgeting
and gives required postoperative care. (PPB) systems, are designed to yield better
Shortly after the operation he, as well as information on the costs of individual pro¬
others in the hospital, receives feedback on grams, where these programs may cross agen¬
the actual condition of the removed appen¬ cy lines.
dix from a pathology laboratory. Parallel Major efforts are also under way to imple¬
feedbacks do not occur with most psychi¬ ment similar accounting systems in state
atric programs. and local governments. In addition, the Fed¬
Evaluating the effect of psychiatric inter¬ eral Government has increasingly required
vention is clearly more difficult than evalu¬ that cost-effectiveness, or cost-benefit, stud¬
ating the outcome of routine surgical proce¬ ies be conducted to justify expenditures.
dures, for several reasons. The state of the Central to this approach is the philosophy
mental patient, both before and after treat¬ that the costs and benefits of a program are
ment, is not easily describable in measurable related and hence cannot be meaningfully
terms. The outcome of psychiatric interven¬ analyzed in isolation. While these innova¬
tion may be unclear until many months af¬ tions have met resistance, their success is
ter treatment is terminated. Even so, the widely recognized and can be expected to
ability to predict the effects of treatment is influence patterns of mental health bud¬
as important for the psychiatrist as it is for geting.
the surgeon. Ideally, the psychiatrist should
Note on Current Assessment Practices
give his patient estimates prior to undertak¬
ing treatment, perhaps in probabilistic Assessment of psychiatric programs
terms, of what the patient can expect of his should include (1) characterization of a
therapy as well as the associated time and population patients; (2) careful descrip¬
of
cost. tion of a program, including associated
Third, mental health programs obtain costs, designed to change them; and (3) out¬
much of their financial support from com¬ come or effectiveness measures consisting of
munity sources, typically government agen¬ a catalogue of changes that have occurred
cies, that are increasingly demanding more after a given time has elapsed. The changes
justification of funding requests. As one that have occurred may then be evaluated as
measure of the extent of community fund¬ good, bad, or indifferent according to the ob¬
ing, recent government statistics show that jectives of the program. The process of de¬
approximately 23,000 psychiatric patients termining outcome measures should force
are in private mental hospitals and general the evaluation team to reflect upon and com¬
hospitals, as compared with 603,000 in state municate program objectives in operational
and county hospitals, and another 60,000 in terms.
Veterans Administration (VA) hospitals.2·3 Outcome assessment is usually a serious
Hence the vast majority of psychiatric beds weak point in psychiatric evaluation. In their
are government-funded. Simply stated, the book on psychiatric rehabilitation Kandel
government agencies involved want to know and Williams4 summarized outcome criteria
—and have a right to know—what they are used in research as falling into three classes:
receiving for the money they spend. (1) changes in psychiatric traits and status;
The increased pressures for program jus¬ (2) hospital adjustment; and (3) community
tification have been stimulated in recent adjustment, as indicated by such criteria
years by the program evaluation methodolo¬ as level of social adjustment, living arrange¬
gy that Secretary Robert S. McNamara and ments, community activities, vocational ad¬
his staff have introduced into the US Depart¬ justment, supervision needed, readmission
ment of Defense since 1961. In particular, we rates, and time out of the hospital. Other
can cite the success of new accounting sys¬ categorizations of outcome assessment have
tems designed to yield better information on appeared in the literature. The operational
program costs, and the presidential directive definition of these diverse criteria remains
that such systems be implemented in most fluid and an active area of research.

Downloaded From: by a Tulane University User on 01/11/2019

 We argue that economic factors should be
added to the list of relevant criteria. Al¬
though economic factors are important in
program selection, rarely have they been
explicitly evaluated along with clinical con¬
siderations. Indeed clinical and economic
factors frequently have been treated as inde¬
pendent of each other even though the
availability and proper use of economic re¬
sources (eg, personnel, physical plant, etc)
are determinants of clinical success.
Relevance of Economic Criteria for
Evaluating Mental Health Programs
The economic effects of alternative men¬
tal health programs need to be analyzed in
conjunction with traditional clinical out¬
come measures if these programs either use
or affect economic resources. Three forms of
economic effect are identified here.
The first effect is the obvious one of the
cost of conducting various programs. Conley
et al5 estimate that $4 billion is spent annu¬
ally on treating or preventing mental illness,
including both inpatient and outpatient
care. Government expenditures account for
two thirds of this $4 billion.6 Hence the pro¬
gram cost, whether or not it is borne by the
patient, should be one criterion for evaluat¬
ing the program. Indeed if we consider the
situation where a fixed amount of money
has been allocated to mental health pro¬
grams, a high per-patient cost implies that
fewer persons can undergo treatment.
The second economic effect is that of the
change in earning power of the patient. This
change is a measure of the increase in the
patient's economic productivity. Many psy¬
chiatric programs, along with other health
programs, should benefit the patient as an
earner and this benefit should be estimated.
A sick person is rarely as productive as he
would be if he were healthy. Consequently,
he depletes his own financial resources or
depends on support from society at large. In
this regard, the annual loss in marketable
output is estimated at $14.3 billion.5 If one
of the objectives of a mental health program
is to permit the patient to be an economical¬
ly independent and self-sustaining member
of society, then one should view as a pro¬
gram benefit the increase in patient earnings
Downloaded From: by a Tulane University User on 01/11/2019
that results from the program. The in¬
creased earnings are taken to represent the
patient's increased economic contribution to
society, and will in practice generally be
shared between the patient and society
through decreased welfare payments, in¬
creased tax payments, etc.
The third economic effect is the financial
loss resulting from antisocial behavior of the
mentally ill. Emotionally disturbed persons
are more prone than stable persons to en¬
gage in illegal or irresponsible activities
such as violence, drinking, promiscuity, and
other socially disruptive activities. They are
also less likely to maintain solid family rela¬
tions. Such undesirable behavior has both
economic and noneconomic adverse effects.
The economic cost of mental illness re¬
sulting from antisocial behavior is difficult
to estimate, and hence the economic benefits
that should be attributed to a particular pro¬
gram because of the reduction of such
behavior through psychiatric care must usual¬
ly be treated as unquantifiable. The dif¬
ficulties in quantifying these benefits do not,
however, make them unreal.
If the concept of the patient as earner is
legitimate, then an ethical problem must be
raised. It is generally accepted that medical
resources are sufficiently scarce that it is not
possible to provide all sick people with the
best medical care. Assume that two individ¬
uals are identical in all respects except their
expected life span, eg, one patient is 25, the
other is 55 years old. The potential for in¬
creasing the earnings of the 25-year-old pa¬
tient exceeds that for the 55-year-old pa¬
tient. Do we therefore conclude that the
25-year-old patient should be treated before
treating the 55-year-old? The decision of
whom to treat first involves a value judg¬
ment that must be made either explicitly or
by default.
Cost-Effectiveness Model as an Approach
to Program Evaluation
In practice, a variety of criteria, both eco¬
nomic and noneconomic, is likely to be rele¬
vant for program evaluation. The problem
remains to develop a framework for compar¬
ing psychiatric programs, given that more
than one criterion is considered relevant. A
cost-effectiveness model, originally discussed

Fox7 in another context, is presented for
by
this purpose.
Two individuals are referred to—the pro¬
gram evaluator and the decision maker.
This clear-cut division does not exist in
practice; the roles of the program evaluator
and the decision maker invariably overlap.
The distinction is useful, however, for pur¬
poses of exposition. In government agencies,
the decision maker is assumed to be a politi¬
cal figure who seeks to allocate funds wisely.
An essential concept of cost-effectiveness
analysis is that of the schedule, which we de¬
scribe in terms of a graph showing the effec¬
tiveness and associated cost of alternative
programs. The program evaluator estimates
the cost that would be incurred and the
effectiveness—or outcome—that would re¬
sult if each of the alternative programs were
undertaken. Again, a simplifying assump¬
tion is made for purposes of exposition, ie,
that there is one operational outcome meas¬
ure. In practice, several outcome measures
are likely to be relevant and must be exam¬
ined individually.
The program evaluator normally may dis¬
card a program that both costs more and is
less clinically effective than another pro¬
gram. On the other hand, if one program
costs more but is more effective than anoth¬
er, the program evaluator should present
both programs to the decision maker to de¬
termine whether the additional effectiveness
is worth the additional cost. One task of the
program evaluator then is to determine
which programs need not be considered by
the decision maker because there are alter¬
native programs that either cost no more
and are more effective or cost less and are at
least as effective. The programs that are not
Downloaded From: by a Tulane University User on 01/11/2019
eliminated from consideration form the sched¬
ule of efficient alternatives.
For example, suppose that a choice must
be made between three alternatives to han¬
dle neurotic patients who do not require
hospitalization and these alternatives are, in
order of increasing cost: (1) to provide no
treatment, (2) to provide outpatient care,
and (3) to provide hospitalization. Let us
assume that the program evaluator judges
outpatient care to be more effective than
hospitalization in addition to being cheaper.
Hence the alternative of outpatient care
dominates hospitalization, which can be
eliminated from consideration. However, the
program evaluator must present the first two
alternatives (no treatment vs outpatient
treatment), which form the schedule of
efficient alternatives, to the decision maker.
The decision maker must then make a value
judgment of whether the benefits of outpa¬
tient treatment outweigh the additional cost
that would be incurred.
The Figure illustrates for formation of a
schedule under the assumption that seven
alternative programs are being considered.
The program evaluator has estimated the
cost of each of seven programs, labeled At
through A7; he has also estimated the effec¬
tiveness of each program. He now seeks to
derive the schedule, assuming that the only
information he has concerning the prefer¬
ence of the decision maker is that, for pro¬
grams with equal costs, the most effective
program is desired and, for that equally
effective programs, the lowest cost one is de¬
sired.
Programs are compared pair-wise to de¬
rive the schedule. Alternative As need not
be considered further since alternative A2
both costs less and is more effective. Thus
A¡¡ is said to dominate A,, ie, it is superior
to A, based on the two criteria that are as¬
sumed to be relevant. It can be noted that
As is also dominated by At and As. Similar¬
ly, Ai need not be considered further since
system A5 is clearly superior. If, after elimi¬
nating As and Aif the remaining combina¬
tion of pairs are examined, the program
evaluator will be unable to find two alterna¬
tives such that one is both more effective
and costs less than the other. The remaining
five then are members of the schedule of
effective alternatives and should be presented
 to the decision maker. These are represented
by dots surrounded by circles in the Figure.
Consider the typical situation where more
than one measure of effectiveness or out¬
come is relevant. Suppose, for example, that
length of stay out of the hospital and a
measure of social adjustment are selected as
appropriate measures. Then the graph in the
Figure would have three axes, representing
the three criteria: cost, length of stay out of
hospital, and social adjustment. If cost is
considered irrelevant or is identical for all
programs being compared, then the cost axis
need not be considered, and we are back to
the two-dimensional case for our two out¬
come measures.
In summary, the function of the program
evaluator is to analyze programs in terms of
the criteria of interest and to make known to
the decision maker his analysis of each of
the dominant alternatives. The determina¬
tion of the best program involves value judg¬
ments unless one program performs as well
as all other programs, based on all relevant
criteria, and is superior, based on one
more criteria.
Some Key Issues in Program Evaluation
Program evaluation was described in the
introduction as having three major compo¬
nents: (1) administrative and financial, (2)
clinical, and (3) treatment outcome. This
section discusses some key issues that
should be specifically addressed by persons
concerned with all three components. These
persons need to develop a greater degree of
sophistication about the interrelationships
involved. The issues addressed are those of:
(1) identification of the population that can
benefit from a program; (2) setting of prior¬
ities on the availability of psychiatric care;
(3) consideration of the effect of funding
procedures on performance; (4) the effect of
bias, and (5) the influence of the adminis¬
trative structure. Since, in practice, program
evaluation involves a strong element of art
and judgment, the importance of these is¬
sues and the best way of handling them will
vary from program to program.
An important question is that of deter¬
mining the patient population for which a
particular program is best suited. Consider
two competing programs, labeled A and B.
Downloaded From: by a Tulane University User on 01/11/2019
It may be misleading to conclude that one
program is better than the other. Program A
may be more effective for one patient popu¬
lation, and program for another patient
population. A major issue is deciding how
the patient population should be stratified,
eg, by demographic variables (sex, socio-
economic level, age, etc) or by diagnostic
variables (schizophrenic, alcoholic, etc).
One practical problem in empirical research
is that there may be a trade-off between a
precise and narrowly defined subsample and
the statistical significance of the results be¬
cause of the small size of the subsample. For
example, suppose that a program serves
both alcoholics and nonalcoholics. Possibly
only nonalcoholics are materially aided by
the program. However, the subsample of
nonalcoholics may be sufficiently small that
the results lose statistical significance if only
nonalcoholics are considered.
Related to this is the issue of establishing
priorities for who should be treated, and at
what level of intensity. If we admit the
or existence of a shortage of adequately trained
mental health workers, a mechanism must
be established to determine how economic
resources (eg, personnel, hospital beds)
should be distributed. The concept is not
pleasant. The matter of priorities has been
handled in our society more by partial solu¬
tions than by overall design. In effect, the
present method of allocating psychiatric
care favors certain categories of persons, in
particular, upper socioeconomic classes and
special groups, eg, veterans or persons em¬
ployed by organizations having particularly
broad health insurance plans. One can ques¬
tion both the wisdom of the current deci¬
sion-making process and the equity of the
resulting system of priorities.
If we accept that medical resources are
limited, the question is not whether a priori¬
ty system is to be established but how. Es¬
tablishing the priority system involves value
judgments that are likely to be beyond the
competence of someone evaluating a partic¬
ular program. These judgments often re¬
quire national policy decisions, either im¬
plicitly or explicitly. It should, however, be
within the competence of the evaluator (or
evaluation team) to describe the population
that will benefit from a particular program.
The effects of the evaluation criteria and
 funding procedures on both individual per¬ vated bias. This type of bias occurs when the
formance (eg, that of a ward chief in a hos¬ researcher, motivated by personal advance¬
pital) and on organizational units need to ment, reports results such that the evidence
be considered. Ideally, measures should be in favor of a particular point of view is
formulated that can serve to evaluate men¬ stressed, and evidence to the contrary is
tal health workers as well as programs. played down or ignored. For an article on
These measures must be carefully designed, how such bias has influenced both research
since the effectiveness of a psychiatrist is and policy decisions in another field, that of
more difficult to assess than that of a sur¬ water resources development, see Marshall.9
geon who performs repetitive and well- The second type arises from the research¬
defined operations. The effect of these meas¬ er's bringing to bear a particular point of
ures on the performance of the psychiatrist view arising from his background or train¬
must be considered, and the measures must ing. Thus a narrow view of a program may
be designed to improve and not degrade pa¬ cause outcome evaluation itself to be atom¬
tient treatment. If the criteria are clearly ized. Even within reasonably well-defined
stated in operational terms, the danger ex¬ disciplines, individuals have their own slant
ists that the person who is being evaluated on what is important. The bias can become
or who identifies with a program being more apparent when we consider the whole
evaluated will avoid difficult patients, ignore gamut of the social sciences in which one
his intuition to the detriment of the patient, person cannot be fully versed. For example,
or otherwise manipulate the statistics. Nor Freeman and Simmons10 discuss how their
is it likely that a system can be designed sociological approach results in their under¬
which eliminates a strong subjective element playing treatment influences in their exten¬
in evaluating personnel. sive outcome study of chronic mental pa¬
The procedures for allocating funds to in¬ tients. On the other hand, others11 more
dividual VA hospitals demonstrate how the within the medical tradition might reason¬
criteria for funding to organizational units ably report the effects of drugs on rehospitali-
can affect treatment. The VA had for years zation rates without closely scrutinizing so¬
funded its mental hospitals solely on the ba¬ ciological influences. Studies attempting to
sis of the size of the hospital census, referred assess the relative importance of variables
to as average daily patient load (ADPL). drawn from diverse traditions are less fre¬
This practice tended to discourage these quent. One exception is the work of Pasam-
hospitals from discharging patients who no anick et al,12 that studies both drug and so¬
longer needed hospitalization. Not until cial context variables for their influence on
fiscal year 1968 were changes instituted so outcome.
that a variety of bases serve to justify fund¬ The final issue relates to the administra¬
ing, including turnover rate, variety and so¬ tive structure of most hospitals. Typically
phistication of programs, medical school there is little communication between the
affiliation, and amount of education and administrators, who formulate the budget,
training conducted. The new allocation and ward chiefs, who use the hospital's re¬
practices were designed to encourage indi¬ sources. This gap is due to several factors,
vidual hospitals to take actions that are including organizational structure, differ¬
more congruent with good patient treat¬ ences in language between psychiatrists and
ment. However, the criteria for funding still administrators, and lack of outcome informa¬
relate only to input rather than outcome. tion. If outcome information were available,
Thus additional analysis is required to de¬ the hospital management could use it for
termine whether the new funding or alloca¬ purposes of budgeting, and a ward team
tion procedures do achieve the desired goals could use it to evaluate its effectiveness and
of patient treatment outcome. to attempt improvements. For example, the
Bias is an issue in evaluation studies and work at Fort Logan, Colo,13·14 provides a
may take many forms. Two types of bias are preliminary investigation of the problems of
described here. The reader is referred to Ro¬ collecting and distributing outcome informa¬
senthal8 for a more complete discussion. tion to different areas in the organization
The finit type is selfish or politically moti- structure.

Downloaded From: by a Tulane University User on 01/11/2019

 Challenge of Program Evaluation
One fruitful area for study is that of de¬
signing patient records so that outcome in¬
formation would be analyzed as a matter of
course. This information would be used to
evaluate psychiatric programs in hospitals
and other settings for community psychi¬
atry. Data on patients would be collected at
set intervals, both during treatment and for
several years afterwards. Such a scheme is
technically facilitated by the development of
computers and related equipment. The ex¬
tent to which evaluation can become a rou¬
tine part of mental health programs is a vi¬
tal issue.
Good evaluation of psychiatric programs
requires the marshaling of skills that have
not been marshaled in the past. Depending
on the nature of the program being studied,
professionals with a knowledge of such fields
as economics, urban affairs, computer tech¬
nology, and systems analysis are able to
contribute to the evaluation. Mental health
workers will find value in gaining acquaint¬
ance with these disciplines so that they can
References
1. Daniels, D., and Kuldau, J.: Marginal Man,
the Tether of Tradition, and Intentional Social Sys-
tem Therapy, Community Ment Health J 3 (1):13-
20 (spring) 1967.
2. US Public Health Service, Department of
Health, Education, and Welfare: Veterans With
Mental Disorders Resident in Veterans Administra-
tion Hospitals, October 31, 1962, Washington, DC:
Public Health Service Publication No.1223,1966.
3. US Public Health Service, Department of
Health, Education, and Welfare: Patients in Mental
Institutions, Washington DC: Public Health Service
Publication No.1452, 1966, parts II and III.
4. Kandel, D.B., and Williams, R.: Psychiatric
Rehabilitation, New York: Atherton Press, 1964.
5. Conley, R.W.; Conwell, M.; and Arrill, M.B.:
An Approach Measuring the Cost of Mental Ill-
to
ness, Amer J Psychiat 124:755-762 (Dec) 1967.
6. Conley, R.W.; Conwell, M.; and Arrill, M.B.:
Estimate of Amount and Distribution of Current
Cost of Mental Illness, 1966, mimeographed.
7. Fox, P.: A Theory of Cost-Effectiveness for
Military Systems Analysis, Operations Res 13
(2):191-201 (March-April) 1965.
Downloaded From: by a Tulane University User on 01/11/2019
communicate and interact with these other
professionals involved.
Throughout this paper the need for more
comprehensive program evaluation has been
stressed. It should be emphasized that im¬
proved measurement is intended to assist
and improve the judgment of those who
make decisions and not to supplant it. In the
final analysis, program selection and organi¬
zation involves value judgments.
Summary
A framework is presented for evaluating
psychiatric programs under the assumptions
that several criteria are relevant. Psychiatric
program evaluation is viewed in the context
of choice, requiring consideration of three
major interrelated activities that influence
treatment and its availability. These are:
(1) administrative and financial activities,
(2) clinical treatment, and (3) assessment
of treatment outcome. Certain key issues in
program evaluation are discussed.
Both authors are participants in research involv¬
ing the application of new concepts to community
psychiatry funded bv National Institute of Mental
Health grant No. MH 02332-02.
8.Rosenthal, R.:Experimenter Effects in Behav-
ioral Research, New York: Appleton-Century-Crofts,
1966.
9.Marshall, H.:"Politics and Efficiency in Water
Development," in Kneese, A., and Smith, S.(eds.):
Water Research, Baltimore: Johns Hopkins Press,
1966, pp 291-310.
10. Freeman, H.E., and Simmons, O.: The Mental
Health Patient Comes Home, New York: John Wi-
ley & Sons, Inc.,1963.
11. Engelhardt, D.M., et al: Phenothiazines in
Prevention of Psychiatric Hospitalization, Arch Gen
Psychiat 16:98-101 (Jan) 1967.
12. Pasamanick, B.; Scarpitti, F.; and Dinitz, S.:
Schizophrenics in the Community, New York: Ap-
pleton-Century-Crofts, 1967.
13. Polak, P.: Unclean Research and Clinical
Change, Milbank Memorial Fund Quarterly XLIV
(1) pt 2, pp 337-345 (Jan) 1966.
14. Binner, P.: Development of the Research De-
partment, Milbank Memorial Fund Quarterly XLIV
(1), pt 2, pp 313-319 (Jan) 1966.