TELEMEDICINE JOURNAL

Volume 1, Number 1, 1995

Mary Ann Liebert, Inc., Publishers

Perspective
On the Definition and Evaluation of Telemedicine

RASHID L. BASHSHUR, Ph.D.

ABSTRACT

Issues related to the definition and evaluation of telemedicine are articulated as a basis for
conducting theoretically based, empirically sound, and policy-relevant evaluation. This pa-
per includes a proposed operational definition of telemedicine, a discussion of the role of
telemedicine in the healthcare system and economic analysis of telemedicine, an analysis of
the basic approaches and requirements for telemedicine evaluation, and an identification of
basic issues for evaluation. Telemedicine is conceived of as an integrated system of health-
care delivery that employs telecommunications and computer technology as a substitute for
face-to-face contact between provider and client. It has the potential for ameliorating seem-
ingly intractable problems in healthcare such as limited access to care among segments in the
population—especially the geographically disadvantaged—uneven quality of care, and cost
inflation. Its true merit has yet to be determined by systematic empirical study. Such study
should include a clear and precise identification of inputs and outputs and the nature of the
relations between them, an assessment of the changes that might occur in the process of care
as a consequence to telemedicine, and, ultimately, an evaluation of the effects of telemedi-
cine on the healthcare system in terms of cost, quality, and accessibility. Several basic ques-
tions regarding the effects of telemedicine are posed as potential hypotheses for future
research.

INTRODUCTION to be resistant to solution, as well as by the

AN AUSPICIOUS START and much
AFTERpromise, more than two decades of rela-
tive calm have prevailed since the early devel-
opment of telemedicine systems in the United
States. Now, the field is reemerging in force,
buttressed by significant advances in technol-
ogy unaccompanied by substantial increases—
indeed, in some instances, decreases—in cost,
Telemedicine also is being revitalized by prob-
lems in healthcare delivery that have proven
for
push developing the national information
infrastructure and the potential for cost shar-
ing with other human services. The promise of
telemedicine that was heralded in the early
1970s for redressing problems of maldistribu-
tion in medical resources, uneven quality, high
cost, and limited access to care has not been
veritably realized. All early demonstration
projects (a total of 16 were funded by the
Department of Health, Education and Welfare
(now DHHS), the National Science Founda-

Department of Health Management and Policy, School of Public Health, University of Michigan, Ann Arbor, MI
19
20
tion, the National Aeronautics and Space
Administration, the former Office of Economic
Opportunity, and the Commonwealth of
Puerto Rico) folded before any definitive con-
clusions could be drawn about the specific
niche of telemedicine in the healthcare system
or its real effects on that system. To be sure,
some of the components of telemedicine, such
as facsimile transmission, teleradiology, and
telemetry, have become widely available, and
telephone consultations long preceded even
the initiation of telemedicine.
Strangely enough, those who toiled earlier to
convince skeptical health policymakers and re-
luctant providers of the potential merit of
telemedicine systems have been vindicated,
not so much by virtue of their eloquence, sound
logic, or positive scientific findings, but rather
by the sheer persistence of serious problems in
healthcare whose amelioration, if not resolu-
tion, still promises to be amenable, at least in
theory, to telemedicine. Indeed, the medically
underserved areas of the past have not become
adequately supplied with the doctors, nurses,
and advanced medical equipment they need.
Confined populations have not become ade-
quately provided with the services necessary
to treat their illnesses and maintain their
health. Clinical information and expertise have
not become redistributed geographically to
match the needs of remote providers and com-
munities within states, regions, the country as
a whole, or, even more widely, the world. And
the system of healthcare has not been restruc-
tured for maximal efficiency, responsiveness,
quality, and cost consciousness. While these
and related problems in healthcare remain beg-
ging for solution, several components of the
requisite technology to enable remote diagno-
sis, treatment and follow-up of patients, as well
as remote education for both providers and pa-
tients, have been improving substantially in
quality and, in some instances, decreasing in
price. It was, therefore, only a matter of time
before telemedicine would be rediscovered,
pressed into service, and ushered into its
proper niche in the healthcare system.
As we embark on developing second-gener-
ation telemedicine systems in the United
States, common sense dictates that we chart a
BASHSHUR
clear course consistent with our nationalgoal
to meet the healthcare needs of the people eq-
uitably regardless of their economic or social
status or their geographic location. In this con-
text, the experience of the first generation of
telemedicine projects must be reviewed for
pertinent lessons to be learned and pitfalls to
be avoided. What was missing then and should
be determined now are the appropriate envi-
ronments for telemedicine systems, the design
of optimal system configurations (including
technology, services, manpower, and organi-
zation) that match the level of needs and re-
sources in communities, and the analysis of
their direct and indirect effects on the provi-
sion of health services and on the communities
where they are located.
This paper has three purposes: (1) to propose
an operational definition of telemedicine sys-
tems to serve as the basis for planning, imple-
mentation, and evaluation; (2) to describe the
anticipated role of telemedicine in the health-
care system and the basic problems in health-
care that are amenable to telemedicine; and (3)
to develop a general framework for evaluating
telemedicine systems. Hence, the paper starts
with a discussion of the definition of telemed-
icine and why it is important to have a clear
and precise definition. Subsequently, it de-
scribes the range of problems in healthcare that
can be addressed, at least potentially, by
telemedicine systems. Finally, a general frame-
work is proposed for comprehensive evalua-
tion of needs and systems.
DEFINITION OF TELEMEDICINE
Despite the fact that telemedicine has been
talked about and tried for more than three
decades and a substantial national investment
in research and demonstration projects has
been made, we have yet to reach consensus on
what it is and what it is not. What is still re-
quired is a clear specification of parameters
and demarcation of boundaries to establish
who (or what) belongs and who (or what) does
not. Without such specification, it would not
be possible to ascertain the true effects of these
systems or the real differences between tele-
TELEMEDICINE DEFINITION AND EVALUATION
medicine and alternative arrangements in
healthcare delivery. However, the definition
should not be arbitrary nor idiosyncratic.
Moreover, it need not be based on consensus
lest it become the lower common denominator
among currently operating programs. Instead,
the definition should be based on rational
(preferably optimal) criteria consistent with so-
cietal goals, technological capabilities, potential
utility, ultimate use, and requirements for ef-
fective clinical practice and information ex-
change.
The common thread in all definitions of
telemedicine (literally, medicine at a distance)
to date is the geographic separation between
two or more interactants engaged in healthcare,
be they provider and client, provider and
provider, or either provider or client and com-
puter. In 1971, Bird,1 the first pioneer to de-
velop a complete prototype system in Boston,
viewed telemedicine simply as "the practice of
medicine without the usual physician-patient
confrontation . .. via [an] interactive audio-
video communications system." Later, in 1975,
he reassured the medical profession that
"telemedicine does not replace the physician or
relegate him [or her] to a less important role
...
[rather], it offers him [or her] a new way to
practice medicine."2 A somewhat broader def-
inition was offered by Willemain and Mark3 as
"any system of care in which the doctor and
his [or her] patient are at different locations."
However, both of these definitions limit the
purview of telemedicine to remote patient care.
A more inclusive concept, telehealth, was in-
troduced by Bennet and his colleagues4 at
Mitre Corporation in 1978; they defined it as
"... systems [to] support the health care
process by providing the means for more ef-
fective and more efficient information ex-
change." They suggested that these telehealth
systems would provide "a broader range of
health-related activities, including patient and
provider education and administration, as well
as patient care." During the 1980s a few "pub-
lic" health educational programs claimed the
title of telemedicine, a misnomer because of
their limited scope, namely, one-way audio
programming of educational materials to the
public. This idea is being reactivated in the
21
form of interactive video programs for con-
sumer health education, a more apt application
for telemedicine.
Telemedicine is conceived of here as an inte-
grated and complete system of healthcare de-
livery and education that is positioned to exploit
the available technological, organizational, and
systemic capabilities. However, the intent is not
to create arbitrary distinctions between what
counts as telemedicine and what does not.
Rather, the purpose is to ensure that such sys-
tems are designed optimally to maximize the ef-
ficiency and effectiveness that may be inherent
in telemedicine. The emphasis on integrated sys-
tems may suggest ways by which the techno-
logical capability of telemedicine is effectively
utilized within a given regional organization or
between institutions within a region on a cost-
sharing basis. In fact, the assumption here is that
there are serious problems in healthcare deliv-
ery that are amenable to technological and or-
ganizational (or systemic) solutions inherent in
telemedicine, but the full benefits may not be re-
alized without developing fully integrated sys-
tems. In other words, telemedicine has the po-
tential to restructure the system of providing
patient care, continuing medical education, and
patient health education, but it can only do so if
it is implemented in full fidelity.
The essential characteristics of telemedicine
systems have been identified5 as follows: (1) the
geographic separation between provider and
client during the clinical encounter (telediag-
nosis) or between two or more providers dur-
ing a consultation (teleconsultation); (2) the use
of telecommunication and computer technol-
ogy to enable, facilitate, and possibly enhance
the interaction between provider and client (or
provider and provider) as well as the transfer
of information; (3) appropriate staffing to per-
form all the necessary functions within such
systems; and (4) the development of an orga-
nizational structure uniquely suitable for im-
plementing telemedicine systems. Two addi-
tional parameters were considered important
for maximizing the effectiveness of telemedi-
cine systems, namely: (5) the development of
clinical protocols for triaging clients to appro-
priate diagnostic and treatment sources; and (6)
the development of normative standards of be-
22 BASHSHUR

havior to replace the norms of face-to-face con- Viewed in this broader context, telemedicine
tact between client and provider (or provider systems constitute a viable response to promi-
and provider). nent and interrelated problems in the delivery
of personal health services in the United States,
namely, equity of access to healthcare, cost con-
TELEMEDICINE AND THE tainment, and uniform quality. Moreover, de-
HEALTHCARE SYSTEM spite the fact that much of the current empha-
sis in telemedicine development is cast in the
The analysis of telemedicine's potential role context of medically underserved rural areas
in the healthcare system and the evaluation of and institutionally confined or physically
its impact on the clients, providers, and society handicapped populations, its ultimate future
at large can be conducted meaningfully only if may well depend on its functional utility for
the concept is clearly defined in precise opera- the general population as whole. Indeed, it is
tional terms, consistent with the six characteris- difficult to lose sight of the fact that the current
tics described earlier. Hence, it is herein proposed configurations of telemedicine systems rely
that telemedicine systems are technologically heavily on tertiary-care medical centers to con-
based innovative systems for the remote deliv- trol, coordinate, and integrate their services
ery of personal health services, continuing med- and, most importantly, to provide the clinical
ical education, and patient health education. expertise for diagnosis, consultation, and re-
Interactive telecommunications and computer ferral. Soon, these centers would realize that if
technology are utilized to facilitate the develop- telemedicine is effective in reaching rural geo-
ment of integrated healthcare organizations (re- graphically remote populations, its benefits can
ferred to as telemedicine systems) for remote, be extended to a much larger urban clientele
multisite delivery of clinical services and edu- living closer but suffering from the same med-
cation. Eventually, telemedicine services will be ical deprivation as their rural counterparts.
distributed widely through freestanding or mo- Although the three issues of access, cost, and
bile equipment that can be placed in people's quality are theoretically separable from each
homes and in a variety of other locations. other, they are, for all practical purposes, in-
Optimal telemedicine systems are designed tertwined: a change in one is likely to produce
to use efficient configurations of manpower, changes in the others. For example, if distance
services, technology, and organization to meet barriers to healthcare are eliminated or reduced
specified community needs, preferences, and via interactive video and audio communica-
economic resources. Such systems should uti- tions, other things being equal, the demand for
lize explicit clinical protocols for patient care, care should increase. Pent-up demand will be-
medical referral, and consultation; and they come manifest wlïen people who have post-
should enable reliable, effective, and fast meth- poned or foregone needed health services are
ods for professional interaction and informa- faced, by virtue of telemedicine, with ready ac-
tion transfer. Therefore, a strong case can be cess to these services. The more health services
made that telemedicine systems should not be they utilize, the greater the cost because the to-
viewed simply as the augmentation of existing tal cost of care is the product of quantity of use
medical practices with telecommunications and unit price. Total cost may enlarge even
and computer technology, but rather as trans- when unit price decreases if the reduction in
formations of existing arrangements that ex- price is offset by an increase in volume. Also,
ploit technological and organizational capabil- there are essential relations between cost and
ities to develop new systems of care. This quality, even though not always one way or di-
statement does not, however, imply that lim- rectly proportional. That is, while higher qual-
ited clinical applications of the technology, as ity typically requires additional cost, more cost
in teleradiology and telepathology, are of no does not assure higher quality.6
use. Rather, these specific applications alone The interconnectedness of these issues is
are not tantamount to complete telemedicine vividly illustrated in recent national efforts at
systems. healthcare reform with the tripartite goals of as-
TELEMEDICINE DEFINITION AND EVALUATION 23

suring universal access to care while limiting in- out telemedicine, on the health status outcomes
creases in cost and maintaining quality of care. of individuals and populations. Both inputs
Irrespective of the merit of the proposed solu- and outputs are then translated into costs (in-
tions, the framers of those reform plans real- cluding time costs). These services consist of a
ized the necessity of addressing all three prob- specified set of inputs involving a level of med-
lems simultaneously. They did not want to ical expertise, medical technology, facilities,
solve the major problem, access to care, only to service personnel, and the clients' preferences
create or seriously aggravate the other two and characteristics.
problems. In that sense, telemedicine may be What is relevant here is that the introduction
the
unique having potential introducing
in for of telemedicine may well change both the care
low-cost, high-efficiency components that may, process and the outputs that result from it. For
under certain conditions, increase access care example, if telemedicine produces more reli-
to
while possibly limiting increases in cost by en- able information for diagnosis than do alterna-
hancing health outcomes. Again, under certain tive conventional methods, we can expect the
conditions, enhanced outcomes may reduce the prescribed treatment to be more appropriate
need for care and, subsequently, utilization of and the health outcomes to be more favorable.
care. Certainly, these are fertile areas for scien- And while it may be feasible to define the in-
tific investigations. puts in a clinically relevant (or theoretically
Speculation aside, if enhanced access to care meaningful) and empirically manageable way,
means anything, under telemedicine or any the definition of outputs remains elusive to
other arrangement, geographic barriers will some extent because the definition of the ulti-
abate and utilization will increase. This change mate outcome, health status, lacks precision.
should increase one of the components of cost. Even if an acceptable measure of health status
The question therefore arises concerning the were devised, the longer the time interval be-
broader economic implications of telemedicine. tween the intervention and the observed out-
The basic economic questions regarding come, the more difficult it is to attribute
telemedicine, as indeed all new arrangements changes in health status to the intervention in
in healthcare delivery, are often stated in terms question. As well, the greater the complexity of
of cost benefit or cost effectiveness. More gener- the intervention, the less able we are to at-
ically, this question relates to the balance in the tribute the observed change to its specific
costs of inputs and outputs. Hence, the eco- source. Added to all this, when we look at the
nomic analysis of telemedicine entails a com- other side of the ledger, the greater the variety
parison of specified sets of inputs and outputs of effects that might be produced, the more dif-
under telemedicine systems versus alternative ficult it is to limit the scope of the evaluation.
arrangements in the provision of personal For instance, the spillover effects of telemedi-
health services. As students of healthcare or- cine on the local economy, the service commu-
ganization have long emphasized, the central nities, and the remote providers of care may
purpose of the healthcare system is to maintain well be of great concern and should, therefore,
the health status of individuals and popula- be incorporated into the definition of outputs.
tions through the treatment of illness and the The environmental/institutional context of
prevention of disease. Nonetheless, health sta- telemedicine systems may have an indepen-
tus is affected by a variety of factors besides dent influence on the type of medicine that is
clinical intervention (or personal health ser- practiced. For instance, prevailing institutional
vices), including human biology, lifestyle, and policies on such things as utilization review,
environment. Yet, the domain of telemedicine preadmission certification, second opinion,
is largely limited to personal health services. and the like, even institutional capacity, may
Economic analysis, therefore, has to limit its fo- well influence utilization, referrals, and con-
cus to assessing the effects of known quantities sultation patterns. These influences may be-
of personal health services (such as physician come more uniform in the future if we develop
visits, episodes of care, hospital stays, pre- regional health networks that link several in-
scriptions, and consultations), with and with- stitutions within one state or region or the
24
country as a whole. Further influences may
emerge in the public sector if new regulatory
agencies evolve. Depending on their form,
function, and authority, new agencies may
simplify or confound some of the issues for
evaluation.
In order to determine the economic effects of
telemedicine, we would need to examine the
outputs of the production process for specified
configurations of technology/services/man-
power/organization. We will need to ascertain
the specific nature of the changes that occur in
the production process as well as the outputs
that are attributable to telemedicine: whether,
for example, a portion of physician time is re-
placed by telemedicine or a non-physician
provider; whether travel costs for client or
provider are reduced; whether a local (and less
expensive) hospitalization substitutes for a
transfer to a remote (and more expensive) med-
ical center; and, perhaps most importantly,
whether clients are more efficiently treated at
the local site without a diminution in the qual-
ity of the care they receive.
Cost-benefit or cost-effectiveness analysis of
telemedicine requires a precise definition of
"fixed" system characteristics in terms of quan-
titative and qualitative capabilities and ex-
pected outputs or objectives. Both approaches
involve the development of a "framework that
is capable of discriminating between their in-
dividual cost, production process, and output.
.. ."7 As suggested by Berki,7 telemedicine may
be only one of a set of systemic changes that
may increase the availability and efficiency of
healthcare in rural areas, including air trans-
portation. Cost-effectiveness analysis will deter-
mine the least costly system that is capable of
delivering a specified set of objectives. Warner
and Luce8 emphasized that the term "cost-ef-
fective" is "a comparative adjective, applied af-
ter assessing a practice's costs and outcomes."
More importantly, they point out "[when] the
benefits of the procedures are not commensu-
rate [or comparable], objective determination
of cost-effectiveness is impossible." Therefore,
in order to be meaningful, the analysis of cost-
effectiveness of telemedicine must be made in
terms of achieving the same set of outcomes in
telemedicine that are obtained by traditional
BASHSHUR
arrangements. Berki7 argued that "the choice of
the most cost-effective means may be an inef-
ficient choice" in view of the multiplicity of ob-
jectives that might be achieved and that "the
cost-effectiveness approach is always one of
sub-optimization." It will be shortsighted to
sacrifice other, more valued, objectives for at-
taining one specified objective at the least cost.
In cost-benefit analysis, a complete listing of all
desired objectives is rank ordered by the least
costly means for their attainment in order to
maximize the benefits for given levels of ex-
penditure. Theoretically, cost-benefit analysis
provides a basis for determining the inherent
worth of telemedicine. Because all costs and
benefits are measured in the same monetary
terms, different programs (e.g., telemedicine
versus traditional arrangements) may be com-
pared in terms of a variety of effects. In cost-
effectiveness, the least costly alternative is de-
termined for a given effect or univariate
program outcome, whereas cost-benefit analy-
sis allows not only the incorporation of multi-
ple outcomes (or benefits) but also comparisons
across very different outcomes (e.g., the bene-
fits of highway construction versus healthcare).
The concerns of program developers are usu-
ally focused on the size of the investment in tech-
nology and operating costs, yet the economic vi-
ability of telemedicine is not likely to be
dependent on these costs, even though they un-
doubtedly are important. When fully utilized,
telemedicine systems may have the unique ben-
efit of being able to address simultaneously the
triple problems of access to care, cost contain-
ment, and quality assurance. If implemented
properly, these systems can bring high-quality
health services to remote, isolated, and under-
served populations, linking isolated providers
with specialists located in advanced medical
centers. In addition, the systems can be designed
with direct (or built-in) mechanisms for cost con-
tainment. Optimally, telemedicine systems
would introduce controls on patients' entry into
the care process on the basis of a formal triage
to direct their care throughout an episode of ill-
ness, a process referred to as a "patient trajec-
tory" by Rockoff and Bennet.9 This approach is
consistent with the current emphasis on man-
aged care as a mechanism for cost control.
TELEMEDICINE DEFINITION AND EVALUATION
In brief, by virtue of their unique capabili- learned. Reflecting
ties, configuration, and design, telemedicine
systems may be able to control the utilization
of service by promoting (or encouraging) the
use of appropriate services (based on an initial as-
sessment of presenting conditions), by appro-
priate providers (nurse practitioner, physician
assistant, or physician), at the appropriate site
(closest to where people live) while discourag-
ing inappropriate use. This way, clients are As-
sured access to appropriate services while they
are denied access to (or discouraged from using)
inappropriate services. At the same time, the
system has the capability of implementing or-
ganizational controls to increase efficiency and
productivity. In their ultimate form, telemedi-
cine systems would bring health services to the
areas where clients live while utilizing a ratio-
nal algorithm for diagnosis, treatment, and fol-
low-up.
The potential effects of optimal telemedicine
systems may be summarized as follows.
Improved access may be achieved directly through
the reduced need for travel and reductions in op-
portunity cost. Improved quality may be achieved
through the adherence to clinical treatment pro-
tocols and the ready availability of consultations
and referrals. Cost containment may be achieved
by substituting lower-cost for higher-cost
providers and facilities, reducing need for care,
increasing benefits, and streamlining the care
process. However, to date, these expectations
have been based on assumptions that have yet
to be verified in a field setting.
EVALUATION OF
TELEMEDICINE SYSTEMS
The evaluation of telemedicine systems is a
complex enterprise because of variations in
definition, system configuration, and potential
outcomes. Hence, it is important from the on-
set to describe the basic requirements for eval-
uation and, if implemented properly, what can
be learned from such an effort.
First, it may help to consider the principal
lessons learned from the evaluation of the first
generation of telemedicine projects, which re-
veal more of what was not done than what was
25
on the evaluation of these
systems through the 1970s, we drew three con-
clusions.10 First, the entire first generation of
telemedicine systems did not fully exploit the
capabilities they had, the extant technological
limitations of the time notwithstanding.
Underutilization was attributed, in part, to the
short-term funding of demonstration projects,
which was confounded by limited institutional
commitment to continue operation beyond the
initial funding period. In other instances,
broadband interactive television and other tele-
metric and support technologies were far in ex-
cess of demonstrable organizational/institu-
tional and community need or the capacity to
use them effectively. Given a condition of un-
deruse, the logical question was whether sim-
pler and lower-cost technology could be more
fully, and therefore more cost-effectively, uti-
lized. Second, nearly every telemedicine project
had narrowly defined functions or special tar-
get groups as their service populations. The
functions included supervision of anesthesia
administration, primary care, radiologie con-
sultation, and psychiatric interviewing, to
name a few. Service populations included hos-
pitalized patients, walk-in patients, nursing
home residents, indigent and remote clients in
isolated Arctic Circle communities, and jail in-
mates. There remains the need for a broad
demonstration of a variety of clinical and edu-
cational applications to a variety of popula-
tions. Third, few firm conclusions were drawn
concerning the specific role (or effects) of
telemedicine in the healthcare delivery system.
Client and, to a lesser extent, provider accep-
tance were rather high, and both tended to in-
crease with experience and familiarity. The
clinical efficacy of telemedicine encounters was
demonstrated to the extent of verifying the in-
tegrity of information transfer in a variety of
clinical functions, including, among others,
dermatologie observations, radiologie consul-
tations, psychiatric interviews, and telemetry.
There was also some evidence that physician
productivity was enhanced under telemedicine
by virtue of utilizing and supervising non-
physician providers at remote sites.10
From a theoretical as well as a policy stand-
point, the evaluation of telemedicine must
26 BASHSHUR

consider appropriate contexts, optimal config- and their outputs in terms of health status, in-
urations, and the full range of effects (immedi- terpersonal relations, and patient and provider
ate/delayed, intended/unintended, direct/in- satisfaction. Other important considerations
direct). Part of the process of developing include resource availability, institutional via-
evaluation designs should be concerned with bility, and the economic stability of the local
assuring that adequate (or optimal) systems are community.
in place. Indeed, three conditions must be met Essentially two types of research questions
before evaluation commences: (1) the identifi- are appropriate for telemedicine evaluation.
cation of the appropriate environments and the The first is biomédical research, which encom-
specific healthcare needs of communities and passes issues of clinical effectiveness and
providers that can be met through telemedicine safety. Specifically, it seeks to ascertain the ac-
systems; (2) the specification of informational curacy, precision, reliability, and sensitivity as
requirements necessary for remote diagnosis, well as the safety of specific technological com-
treatment, and follow-up as well as education; ponents (rather than total systems of care) in
and (3) the attempt to exploit to the extent pos- providing diagnostic and therapeutic informa-
sible the technological and system capabilities tion. The basic question concerns the extent to
that are in place. which specific telecommunications and com-
Optimal evaluations (i.e., those that provide puter systems meet clinical standards of per-
accurate answers to the relevant questions) can formance compared with in-person observa-
be performed only if optimal systems are put tion and measurement. The second type of
in place, lest the evaluation simply reflect im- questions pertains to health services research,
perfections in design rather than true capabil- which focuses on the effects of telemedicine on
ity. Subsequently, rigorous and scientifically health care delivery and its acceptance by the
valid research designs are needed to determine providers and clients.
the effects that telemedicine systems may have From a policy perspective, research to assess
on access, cost, and of
quality care; their ac- the merit of telemedicine only with regard to
ceptance by providers and clients; and their im- current technological capability may be short-
pact on the institutions and communities sighted. Technological development is ex-
where they are installed. The information to be tremely rapid in terms of both quality im-
derived from the evaluation is necessary, not provement and price decline. Technological
only for rendering policy-relevant decisions advances have solved some problems that were
about the ultimate merit of these systems and considered intractable a decade or so ago.
their funding from public or private sources, Additionally, as telecommunications and com-
but also for making design and programmatic puter technology evolves from generation to
adjustments, refinements, and corrections to in- generation, there is~a general decrease in asso-
duce maximal benefits. ciated costs. Rapid changes in technology as
Unlike single medical interventions or new well as project-specific needs and funding may
medications that are amenable to tests of effi- result in idiosyncratic combinations of tech-
cacy, effectiveness, and safety, telemedicine nology at specific sites. Further complicating
systems constitute innovation "bundles" that the situation, many components may become
can be presented in various configurations, and obsolete and need to be upgraded or replaced.
they continue to evolve. It is futile, at least in Nonetheless, research is necessary to evaluate
the short term, to conceive of them as "fixed," the ability of technology to provide the re-
single, and consistent models of healthcare de- quired level of service. Furthermore, selected
livery. Hence, their evaluation has to take into biomédical aspects should be evaluated in net-
account not only the variability in the contexts works that are sufficiently matched to specific
where they are implemented, but also their in- clinical requirements in terms of the level of so-
puts in terms of specific technology/ser- phistication in technology.
vices / manpower / organization configurations The focus, dimensions, concerns, and meth-
as well as patient preferences and characteristics odologies of these two kinds of research for
TELEMEDICINE DEFINITION AND EVALUATION
Table 1. Types of Research
Biomédical
Focus Clinical performance
Dimensions Efficacy
Effectiveness
Safety
Concerns Accuracy
Reliability
Precision
Sensitivity / specificity
Methodology Performance studies
Clinical trials
telemedicine evaluation are depicted in Table
1. In the remainder of this section, we focus on
the evaluation of telemedicine from the per-
spective of health services research. The
process of evaluating telemedicine from this
perspective can be described as consisting of
three consecutive stages: evaluability assessment,
formative evaluation, and summative evaluation.
Evaluability provides
assessment opera- an
tional definition of the system of telemedicine
in place; its specific attributes, characteristics,
and configurations; and, most importantly, a
determination of the specific problems and is-
sues to be subsequently evaluated. Before one
can design a meaningful and valid evaluation
methodology, it is essential to identify all the
inputs and outputs and the nature of the antic-
ipated relations between them. A critical com-
ponent in evaluability assessment, therefore, is
the specification of objectives in terms of the
major stakeholders; i.e., the benefits and costs
from the perspectives of clients, providers, in-
stitutions, and the community at large. Thus,
when done properly, evaluability assessment
will frame the evaluation issues. It sets the stage
for the systematic formative and summative
evaluations to follow.
One simple example of evaluability assess-
ment is the development of a matrix (that may
be two- or three-dimensional) which indicates
the four perspectives of clients, providers, in-
stitutions, and society at large on one axis and
27
for the Evaluation of Telemedicine
Health services
Acceptance and effects on healthcare delivery
Accessibility
Quality
Cost
Perspectives of client, provider, and society
Field studies
Surveys
Field observations
Experimental studies
Controlled
Quasi-experiments
the three types of effects—access, cost, and
quality—on the other. The cells within the ma-
trix would then contain the specific changes
that might be observed by the combination of
perspective and effect. This listing, in turn, can
serve as the basis for developing the hypothe-
ses to be tested in the evaluative study to fol-
low.
Formative evaluation focuses on the
descrip-
tion of the system design and implementation
and, more importantly, on the assessment of its
intermediate- or short-term effects on the
process and content of care. While largely de-
scriptive in nature, this evaluation is useful for
subsequent adjustments and refinements that
may be needed to fine tune the system, and,
perhaps more importantly, for dissemination
activities both nationally and internationally.
More important, however, is the formative as-
sessment of intermediate effects on the system
of delivering health services, as outlined in
Table 2.
Summative evaluation seeks to determine the
ultimate effects of telemedicine systems on
health outcomes, which can be measured in a
variety of ways, including objective and sub-
jective measures, problem resolution, and func-
tional performance.
Finally, the complexity and richness of the
task of evaluating telemedicine systems may be
demonstrated by the variety of questions that
should be answered. Interestingly, the trends
28 BASHSHUR

Table 2. Formative Assessment of Intermediate Effects

Health Care Delivery
Content of care Diagnosis, treatment, follow-up prevention
Process of care Scheduling, waiting time, service time, patient flow, case finding
Intermediate outcomes Outpatient visits, hospital admissions, length of stay
Effects on
Clients Functional status, satisfaction, access to care, knowledge, attitude
Providers Patient load, patient mix, satisfaction, knowledge, attitude
Institution Productivity, efficiency, provider mix
Community at large Availability of health resources, local economy,
cost-sharing with other human services, etc.

in the development and deployment of tech- tems? Is it a function of

nology are irreversible. Hence, there is little
question whether interactive video/audio in-
tegrative communication systems will be uti-
lized in the future in medicine and other hu-
man services. From a policy standpoint, the
important questions to be raised now have to
do with our ability to exploit the developing
technology to meet our basic human needs, in-
cluding healthcare. Once the basic technology
becomes ubiquitous, and it will, researchers
will be busy trying to explain the effects after
the fact. In some sense, the opportunity is now
open to help shape the future, at least insofar
as asking probing questions for evaluation that
may yield useful information for developing ef-
ficient and effective systems of healthcare de-
livery. Examples of such questions should clar-
ify these issues.
Because the barriers of distance and time ex-
ist for a significant portion of the population by
virtue of their geographic separation from ap-
propriate healthcare resources, physical hand-
icap, or institutionalization, a leading question
for study is whether telemedicine systems
would provide a significant improvement over
alternative arrangements in spatial and tempo-
ral access to care for many, and at what cost.
For those persons isolated from mainstream
medical care for one reason or the other, the
question is whether telemedicine would pro-
duce a demonstrable improvement in the qual-
ity of care available to them; whether the avail-
ability of consultations with specialists to
remote physicians would have an impact on
the quality of care that is rendered. What spe-
cific mechanisms contribute to the productiv-
ity of physicians working in telemedicine sys-
having non-physician
providers in remote settings supervised and
supported by physicians in central locations? Is
physician time allocated more efficiently
among multiple delivery sites and at a level
and quality of performance commensurate
with medical training? How does "telemobil-
ity" enable physicians of various specialties to
deliver care over a great distance and to a large
number of different sites?
Particularly in the situation where accessible,
high-quality, and personal primary care has
not been available before, one task is to deter-
mine the extent to which there are interactive
effects among the expectations of clients and
the abilities of providers and the system to meet
them. Of importance are changes in the expec-
tations of patients concerning accessibility of
care, the quality of care rendered, and their
combined effects on expectations of level and
use of health services. There may be a greater
demand on the level and kinds of care made
available. But we have yet to determine the pro-
ductive capacities of telemedicine systems.
Particularly, given a configuration of technol-
ogy / services / manpower / organization, what
are the levels and kinds of demand for health-
care services that can be met, and at what level
of quality?
Reduced distance and time to care may also
have implications for client expectations con-
cerning other access factors. With reduced
travel required, will patient expectations about
reasonable waiting and service times be
higher? One study11 found telemedicine en-
counters to be longer than face to face events.
Did this finding reflect a limitation in the tech-
nology of that era, which is now resolved, or
TELEMEDICINE DEFINITION AND EVALUATION
an inherent limitation of such systems? If the
finding is confirmed by new studies, will it be
perceived as a problem by either clients or
providers?
Related to patient expectations of the acces-
sibility and quality of care received in telemed-
icine systems is the need for investigating the
content of the interaction between the physi-
cian and patient over the medium of telecom-
munications, such as telephone or television.
Whereas we know that patient acceptance of
interactive videoconferencing is high, espe-
cially with added experience and familiarity,
we have yet to determine whether telemedicine
encounters facilitate patients' communication
of the discomfort being felt, the symptoms, or
the anxiety. What is the effect of communicat-
ing instructions for medications or treatment?
Is patient comprehension and compliance
likely to increase or decrease as a result of the
medium of communication?
For the provider, faced with the increasing
requirements to be abreast of technical knowl-
edge and to utilize this knowledge in the treat-
ment of patients, the facilitating aspects of
telecommunications are not well understood.
Does providing access to scheduled clinical
conferences involving remote specialists in-
crease the likelihood that a physician will "at-
tend" these presentations to keep abreast? In
having the facility to consult with remote spe-
cialists, will the physician's consultation and
referral patterns change, and if so, how?
Research is needed on the effects of telemed-
icine on the organizational structure and envi-
ronmental setting in which it is placed. What
are the organizational consequences of the re-
allocation of diagnostic and treatment activities
between the non-physician provider (if in-
volved) or remote physician and the consult-
ing physician at the central location under
telemedicine? Are problems of communication
and control over the conduct of care heightened
or reduced? Also important is whether, when
mediated by a telecommunications system, the
hierarchical relationships between the special-
ist at the center and the remote provider are
changed in any way. We have yet to under-
stand the consequences of introducing new
roles such as the technician (engineer), who is
29
at once part of and apart from the delivery of
care.
If the evaluation of the current generation of
telemedicine systems is to draw accurate con-
clusions about the true merit of telemedicine,
how it fits into the system of healthcare deliv-
ery, and how it should evolve in the future,
these and other similar questions should be an-
swered. Ultimately, the success of the telemed-
icine evaluation will depend on our ability to
conduct scientifically valid, empirically rigor-
ous, and policy relevant research.
ACKNOWLEDGMENT
I wish to express my gratitude to Jay Sanders,
Ace Allen, Gary Shannon, and Ellen Johnson
for reading an earlier version of this paper and
offering useful comments and suggestions; to
Richard Hirth and Dean Smith for reviewing
and commenting on sections of this manu-
script; and to two anonymous reviewers for of-
fering valuable suggestions.
REFERENCES
1. Bird KT. Teleconsultation: a new health information
exchange system. Washington, DC: Third Annual
Report to the Veterans Administration, 1971
2. Bird KT. Telemedicine: concept and practice. In:
Bashshur RL, Armstrong PA, Youssef ZI, eds.
Telemedicine. Springfield, IL: Charles C Thomas,
1975:90
3. Willemain TR, Mark RG. Models of health care sys-
tems. Biomed Sei Instrum 1971;8:9-17
4. Bennet AM, Rappaport WH, Skinner EL. Telehealth
Handbook, Bethesda: US Department of Health,
Education and Welfare, Publication No. (PHS) 79-
3210,1978
5. Bashshur RL, Armstrong PA, Youssef ZI. Telemedicine:
explorations in the use of telecommunications in health
care. Springfield, IL: Charles C Thomas, 1975
6. Donabedian A. Quality and cost: choices and re-
sponsibilities. Inquiry Spring 1988;25:90-99
7. Berki SE. Telemedicine: some economic implications.
In: Bashshur RL, Armstrong PA, Youssef ZI.
Telemedicine: explorations in the use of telecommunica-
tions in health care. Springfield, IL: Charles C Thomas,
1975:181
8. Warner KE, Luce BR. Cost-benefit and cost-effectiveness
analysis in health care. Ann Arbor, MI: Health Admin-
istration Press, 1982
30 BASHSHUR

9. Rockoff ML, Bennet AM. The "patient trajectory": A phone for remote medical consultation. N Engl} Med
modeling tool for planning and evaluating rural 1975;292:722-732
telemedicine systems. Presented at the NATO
Telecommunications Symposium. Bergamo, Italy,
September 1977 Address reprint requests to:
10. Doermann AC, Macarthur DL, Walcoff P. Extending Rashid L. Bashshur, Ph.D.
the capabilities of non-physician providers in isolated
rural areas: an investigation of the potential impact Dept. of Health Management and Policy
of telecommunications based technologies. Mitre School of Public Health
Corporation Report, McLean, VA, October 1975 University ef Michigan
11. Moore GT, et al. Comparison of television and tele- Ann Arbor, MI 48109