You are on page 1of 22

P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ

Applied Intelligence 04-Feret January 6, 1997 14:37

Applied Intelligence 7, 57–78 (1997)


°
c 1997 Kluwer Academic Publishers. Manufactured in The Netherlands.

Combining Case-Based and Model-Based Reasoning


for the Diagnosis of Complex Devices

M.P. FÉRET AND J.I. GLASGOW


Department of Computing & Information Science, Queen’s University, Kingston, Ontario, Canada, K7L 3N6
janice@qucis.queensu.ca

Abstract. A novel approach to integrating case-based reasoning with model-based diagnosis is presented. This
approach, called Experience Aided Diagnosis (EAD), uses the model of the device and the results of diagnostic
tests to index and match cases representing past diagnostic situations. Retrieved cases are then used to overcome
errors created by the application of incorrect device models. The diagnostic methodology is described and applied
to two real-world devices. Experimental results demonstrate the effectiveness of both the indexing schema and the
matching algorithm. The paper discusses how these results can be generalized to multiple fault situations, to other
types of device models, and to other applications in the field of an artificial intelligence.

Keywords: model-based diagnosis, case-based reasoning, hybrid problem-solving methods

1. Introduction Case-based reasoning (CBR) has traditionally been


used as a stand-alone problem-solving method [25, 30],
This paper presents an approach to integrating case- sometimes applied to diagnosis problems (e.g., [41]).
based reasoning with a more traditional diagnostic Only recently has CBR been used in association with
method based on the structural isolation for diagno- other problem-solving paradigms [21, 34, 39]. Our
sis of complex devices. This generic approach to di- approach is novel in that it uses CBR only after the
agnosis is based on a hierarchical decomposition of MBR process has taken place, and in that it uses the
mechanical devices and uses sensor data, collected model and the results of the MBR process to index the
in real-time and stored in a database, to guide the cases.
search towards hypothetical diagnoses [14, 17, 35]. The hybrid CBR/MBR methodology, Experience-
Some of the difficulties encountered while applying a Aided Diagnosis (EAD), described in this paper in-
model-based reasoning (MBR) diagnostic method to corporates a critique of the results of the model-based
two real-world devices are identified. The difficul- approach in the light of past experience and provides
ties arise from imperfections of both the model of the the human operator with a means for exploring alterna-
device and the human experts designing the model. tive solutions. The integration of CBR with the struc-
These imperfections lead to incorrect and incomplete tural isolation process allows for a simple and effective
models which produced inadequate diagnostic perfor- indexing scheme as well as a computationally inexpen-
mance. In this paper, we further develop ideas ini- sive similarity measure for cases.
tially introduced in [14, 15] and in [16] by providing This paper initially reviews alternative approaches to
additional motivations for our approach and extensive automated diagnosis and introduces case-based reason-
experimental results that support the claims of the pa- ing and its applications to diagnostic problems. It lists
per. The experiments described in this paper also pro- some of the problems arising when trying to apply any
vide an example of an evaluation strategy for CBR MBR diagnostic method to complex devices and moti-
systems. vates a hybrid approach combining CBR and MBR.
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

58 Féret and Glasgow

This approach is then presented in detail. Experi- to focus the abductive search with necessary condi-
mental results show the effectiveness of the method. tions. This is equivalent to reducing the size of the
A final discussion summarizes the contributions of this search space for minimal conflicts by compiling some
research. This paper also suggests a possible experi- deep knowledge. De Kleer has tried to focus the truth
mental evaluation method for both “pure” and hy- maintenance systems either with statistical information
brid CBR systems. It also describes a possible tool [10], or failure modes [9] or more refined look-ahead
for the verification and validation of knowledge-based search strategies [11]. Friedrich [18] uses hierarchy
systems. decompositions of the device to improve the efficiency
of his diagnostic algorithm [18]. Mozetic defines ab-
straction operators on the device models, which lead to
2. Model-Based Diagnosis hierarchical diagnosis as well [37].
Another problem with MBD is the difficulty of rep-
MBD has emerged as a relevant research topic from resenting complex device models where possible inter-
the problems found in traditional ruled-based diagnos- actions among components are sometimes overlooked.
tic expert systems. These earlier systems were unable This problem does not arise with logical circuits be-
to handle unpredicted faults, had poor explanation fa- cause the components of such devices are well de-
cilities and did not take advantage of existing design scribed, and their connections and interactions are well-
specifications. MBD addresses these problems by us- understood. Models for complex devices, however, are
ing device models containing “deep” and first-principle not used for diagnosis in the real-world. Models are
knowledge which describe the correct, expected be- also complex in themselves and need validation and
havior of the device. The search for components in an verification. Validating and verifying knowledge bases
abnormal state is guided by the discrepancies between is difficult and little success has been achieved in this
what is predicted by the model and what is observed in area. It is therefore likely that any MBD system will use
the device. A diagnostic session is triggered when ini- an incomplete or incorrect device model. The current
tial symptoms do not match with the predictions of the literature does not account for this extraneous difficulty
model. Some models are fault models; they only de- and is therefore unrealistic.
scribe conditions under which a component or a group Model-based approaches require extensive and accu-
of components is or might be faulty. rate device models. Such models are difficult to design
The main task of diagnosis is to find explanations for for a real-world device including mechanical and elec-
a set of given symptoms. In the context of diagnosis, trical parts, as well as real-time sensors. They often
explanations are conjectures that must either be con- are very complex models of not so complex devices
sistent with the symptoms and the model of the device [7, 26, 39]. Even though the basic components from
or entail the symptoms. Console et al. define a model which these devices are built are usually simple, the
for diagnosis that encompasses explanations that sat- complexity of the association required to assemble a
isfy both of these criteria [8]. This model subsumes device with interesting enough performances usually
Reiter’s [42] and De Kleer’s [12] seminal work. Reiter leads to difficulties for diagnosis [19]. One single com-
describes an algorithm that computes the minimal con- ponent can make a whole device, such as a car or a nu-
flict sets and derives the minimal diagnoses from them. clear reactor fail. It is often difficult to determine the
This algorithm is NP-hard in the general case, in agree- completeness and consistency of a device model. This
ment with Bylander et al.’s results [5]. Reiter’s char- makes model-based diagnostic systems unreliable and
acterization of diagnoses also corresponds to deKleer unapplicable for critical applications where safety is a
and Williams’ definition of diagnosis [12]. Like Reiter, primary issue.
they use the notion of minimal conflict set, computed Analyzing and compiling human diagnostic prob-
with TMSs to derive minimal diagnoses. lem-solving capabilities is difficult. Misunderstand-
One problem with MBD is its inherent compu- ings, incorrect specifications, typos, etc., typically lead
tational complexity. Abduction in MBD is, in gen- to partially incorrect models which are hard to debug,
eral, NP-hard, making diagnosis untractable for even especially when there is no simulation program avail-
medium-sized devices. Many researchers have tried able for the device. Moreover, device models are not
to focus the search for (minimal) diagnoses in MBD always the most natural or efficient representation for
[7, 9–11] or to reduce the complexity of MBD methods diagnosing faulty components [46]. These knowledge
(e.g., [18, 37]). Console et al. use compiled knowledge acquisition and validation problems clearly weaken the
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

Combining Case-Based and Model-Based Reasoning 59

reliability of model-based systems and need to be ad- and adapts them to derive and criticize a solution. If
dressed before the system can be used for critical, real- the solution is not satisfactory, new cases are retrieved
world applications. Fault models consisting of neces- to further adapt it in the light of additional constraints
sary conditions (for parts to be faulty) are simpler to (expressed from the non-satisfactory parts of the pro-
express and to implement than correct behavior models. posed solution) until it is acceptable. After a problem
However, fault models have a well-known drawback: is solved, a new case can be created and stored in the
they only model foreseen, predictable faults. There- case-base. Overviews of the CBR paradigm can be
fore, diagnostic systems based on fault models are in- found in [31, 43, 47].
effective in the presence of unforeseen faults.
The process of human diagnostic problem-solving is
often suboptimal [24, 50]. The following biases, ex- 3.1. Main Challenges in Building a CBR System
tracted from [24], are the most relevant causes of sub-
optimal diagnostic performance: (1) Not all possible The challenges involved in the task of building any
causes are taken into account, (2) Evidence invalidat- CBR system are defining an indexing schema allow-
ing the diagnosis is ignored, (3) Absence of symptoms ing for fast and accurate retrieval, adapting the closest
is ignored, (4) The probability of causes is estimated matching case to the current instantiation of the prob-
incorrectly, and (5) Only confirming actions are per- lem and critiquing the derived solution. While these
formed. These shortcomings, present in human rea- problems do not lessen the relevance and the contri-
soning, are likely to be found in any model designed bution of CBR as a paradigm to automated reasoning
and implemented by humans. In our experience, we and learning, they need to be addressed and solved for
have found such mistakes in the experts’ explanations real-world domains to make CBR a practical problem-
and reasoning processes. This leads to models that are solving methodology.
either incomplete or inconsistent because they incor- One condition of applicability of CBR to a specific
porate human limitations. problem is that there exists an initial case-base. This
is the only way for a CBR system to take into account
prior background knowledge. This is sometimes im-
3. Case-Based Reasoning possible. For example, a diagnostic system that is built
at the same time as the device it is supposed to mon-
Case-based reasoning is a psychological theory of hu- itor cannot be given prior knowledge in the form of
man cognition. It accounts for issues in memory, learn- diagnostic cases. The failure rate of some very reli-
ing, planning and problem-solving. Its foundations able engineering systems can be very low, making the
lie in Schank’s conceptual memory model [44], an in- building of the initial case-base difficult.
tegration of semantic networks [6, 38] and episodic As many researchers have pointed out, indexing of
memory [49]. A key concept of Schank’s model is cases is essential to achieve good performance (e.g., [2,
that knowledge is derived from experience and that a 29]). The goal of indexing is to allow the CBR system
theory of memory must account for knowledge acqui- to efficiently retrieve cases that have the potential to
sition. Schank defines “memory organization packets” make relevant predictions about the new case. Unfor-
(MOPS) [45], which are general structures to account tunately, as the complexity and the size of the problem
for diverse and heterogeneous episodic knowledge. increases, the likelihood of having previously solved
Schank’s work constitutes the basis for case-based rea- a similar problem may decrease and, even if relevant
soning computer systems such as MEDIATOR [33] and cases exist in the case base, the matching cost becomes
PERSUADER [48] for dispute resolution, JULIA [27], prohibitive. The indexing problem presents inherent
and KRITIK [20] for design, CHEF [25] for planning, trade-offs between creativity and relevance, general-
HYPO [1] for legal reasoning, CASEY [34], PROTOS ity and accuracy, coverage and efficiency. Kolodner
[2], and CELIA [40] for diagnosis, LADIES [4] for insists that indices should be predictive of the solu-
decision support. tions to the problem, abstract enough to cover different
CBR has traditionally been used as a stand-alone cases, concrete enough to be recognizable in future
problem-solving method [25, 30]. A CBR system cases and “useful” in predicting relevant parts of rel-
stores past experiences in the form of cases. When evant solutions [31]. These guidelines are somewhat
a new problem arises, the system retrieves the cases contradictory and leave a lot open to interpretation.
most similar to the current problem, then combines The most common methodology for index selection
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

60 Féret and Glasgow

is still a case-base approach: see what has been done Critiquing and adapting cases are also difficult prob-
before for similar problems and try to improve on it, lems. Critiquing involves recognizing the flaws of a
while considering the specificity of the new domain or proposed solution and identifying the causes of these
application. flaws. Adapting involves local knowledge of different
Typically, a subset of the features describing a new parts of cases and of their interactions. The system
case is used as indices into the case-base. This method needs to know what is wrong in a solution with respect
is based on the assumption that cases with similar so- to a specific goal, what are the parts of the case that are
lutions have similar problem statements. In domains deficient and, under the current circumstances (e.g., the
where such an assumption holds (e.g., CHEF’s domain current goal and the current proposed solution) what
[25]), the continuum of problem statements can be needs to be changed to produce a better solution. In
mapped onto a continuum of problem solutions through real-world CBR applications, these two steps are left
a smooth adaptation strategy. This strong assumption to the users.
does not hold in many real-world domains. For exam-
ple, in a diagnostic domain, a small change in a set of
symptoms can lead to a totally different cause. Even 3.2. Hybrid CBR Approaches
under this assumption, defining the right set of indices
is a hard task, which is typically left to the designers. The main issues to address when building CBR systems
Kolodner has described an inductive method to re- are to define an effective indexing schema, efficient re-
duce the index set and generalize the remaining indices trieval and storage mechanisms, a reliable similarity
[28]. Barletta and Mark argue that inductive methods measure for cases and an adaptation mechanism. CBR
can rely only on the features describing the cases. They is limited by the difficulty of indexing, retrieving and
convincingly point out that usually some background evaluating previous experiences. This is especially true
knowledge about the domain could help index the cases in the case of diagnostic applications, where similar
[3]. Their method, inspired by the explanation-based symptoms can have very different or multiple causes.
learning community, uses the explanations of the ob- Techniques that work for small problems do not neces-
servable features to improve the indexing. This index- sarily handle scaled-up versions of the same problem.
ing method is computationally expensive compared to Model-based approaches are limited by the fact that
a simple indexing schema. It also requires a powerful complete and consistent models of complex devices
domain theory (e.g., an adequate device model), as well are hard to produce and that often incorporate human
as a means to select explanations that are relevant to the limitations.
final solution. This may be difficult if many adaptation Researchers have previously combined CBR with
steps are taken to reach a good solution. other problem solving paradigms. Rajamoney and Lee
If more than one case is retrieved from the case- use CBR to decompose a novel, large problem into
base to help solve a new problem, a similarity metric smaller known ones that are then solved with a model-
is usually used to decide which one is the most rele- based reasoning (MBR) system [39]. Thus, they use
vant to the current problem. Similarity may depend on CBR for a separate task than the one the MBR system
context, on inter-related attributes describing the cases, is used for, namely decomposing a large problem into
etc. On a wider basis, similarity metrics also depend smaller ones.
on the indexing process. For example, the ranking of CELIA, a diagnostic system for automobiles, is the
very similar (i.e., if the indexing schema is effective) counterpart of the work of Kolodner and Kolodner [32]
cases might be easier than ranking dissimilar ones. On in a mechanical domain [40, 41]. This work combines
the other hand, if cases do not contain enough informa- explanation-based learning and case-based reasoning
tion to distinguish among them, ranking them might be to model the evolution of a novice diagnoser to an ex-
extremely difficult. pert one. Each diagnostic case is decomposed into
In diagnostic problem solving, the same symptoms snippets that represent only part of the action-decision
do not necessarily lead to the same failures. Evaluat- sequence of the case. Redmond argues that this type
ing the similarity of intermediate reasoning steps seems of representation for cases is more flexible than mono-
necessary, whether they are explanations provided lithic cases because it allows for easy recombination
by the users or tests generated by the system itself. of knowledge chunks as well as direct access to more
Thus, judging similarity based on symptoms is clearly local knowledge. It also makes other problems such
insufficient. as indexing and retrieval more complex. Combining
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

Combining Case-Based and Model-Based Reasoning 61

snippets to create new cases might create compatibility system called the Fairing Servicing Subsystem, and a
and relevance problems. Redmond’s approach is a very Reactor Building Ventilation System [13, 14]. The first
specific one, where learning is continually prompted device is located at the rear of a ship to automatically
by an instructor. Each step of the diagnosis is carefully replace damaged fairings on a cable which drags an
analyzed, explained, criticized, and potentially used underwater detection system. The fairings prevent the
to learn something new. The evaluation of the system detection system from drifting away from the axis of
does not include criteria such as autonomy of behavior, the ship. The Reactor Building Ventilation System is a
accuracy of results, efficiency of the indexing method, modified model of a ventilation system for an existing
learning speed, etc. nuclear power plant1 .
Koton’s CASEY system uses CBR to speed up a
model-based diagnosis system by storing previous ex-
periences and recalling them when appropriate [34]. In 4.1. Diagnosis as a Structural Isolation Process
this system, CBR is used first as an attempt to reason by
analogy with a previously encountered situation. The Within the ADMS framework, EAD is applied in
cases are directly derived from the MBR system and are association with structural isolation, involving fault-
used, once created, in isolation from the MBR system. models. One strength of EAD is that it can be applied
In these two systems, each paradigm is applied inde- in contexts of correct behavior models as well, and
pendently. Golding and Rosenbloom use CBR to im- is therefore applicable to any model-based diagnostic
prove the accuracy of a rule-based system [21]. Their approach.
cases denote exceptions to rules and are indexed by Structural isolation is based on considering the de-
the rules that they confirm and by the rules that they vice as a hierarchy of components or groups of com-
contradict. Cases do not store the same information ponents. This hierarchy is expressed using a frame
as the rules, but provide a different source of informa- language and constitutes the backbone of the knowl-
tion for decision making. These systems all show that edge base built for each application of the ADMS to
CBR does improve the performance of the whole sys- a mechanical device. The diagnosis algorithm starts
tem by either speeding up the primary process without at the top of the hierarchy—the node representing the
bringing new information or by storing a different type device itself. All the recognizable components that
of experiential knowledge that is used to improve the can be diagnosed as sources of failure are situated at
accuracy of the overall system. However, all of these the bottom of the hierarchy. Each basic component
systems have to face critical problems in retrieving and has associated failure mechanism patterns. These pat-
matching their cases, problems that are typical of CBR terns are conjunctions of sensor functions, testing spe-
systems. cific conditions on the sensor data stored in a database
[35]. Between these top and bottom levels, the device is
decomposed into meaningful substructures, associated
4. Experience-Aided Diagnosis with test conditions indicating whether these substruc-
tures are potentially faulty. In this context, performing
Our approach to diagnosis combines case-based rea- diagnosis involves traversing the hierarchy according
soning and model-based diagnosis in a novel way. Ex- to results of necessary conditions applied to the sub-
perience, stored in the form of cases, is used to evaluate structures represented in the hierarchy. At a given node
and critique the results of the model-based diagnos- in the hierarchy, if there is no evidence that the sub-
tic process. The goal of the case-based reasoner is structure can be faulty, then the whole substructure can
to aid in the final selection of the diagnosis. We call be pruned from the search space of potentially faulty
our approach to diagnosis Experience-Aided Diagnosis components. If there is such evidence, the substructure
(EAD). is examined further and more local conditions are ap-
EAD has been fully implemented as part of a generic plied to subnodes of the current node. This process is
diagnostic system, the Automated Data Management known as the structural isolation process [36] and also
System (ADMS), which has previously been described handles multiple faults by simultaneously investigating
and compared to other techniques for diagnosis in [15]. multiple paths in the model.
This section describes the initial diagnostic method Hierarchical, structural and/or conceptual data struc-
used for the ADMS and outlines the problems that tures have already been found useful for diagno-
arose in applying it to two real-world devices, a robotic stic applications [22]. Hierarchical diagnosis is any
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

62 Féret and Glasgow

diagnostic process that is based on a structural abstrac- confidence in each potential diagnosis as well as cri-
tion of the device being diagnosed. The most common teria related to the history of each component (such as
example is a structural hierarchy. Genesereth defines meantime between failures, time to life expectancy).
the general cost of a hierarchical diagnosis to be:
X
H 4.2. The Case-Based Component of the ADMS
Fi (Ni )
i=1
The EAD hybrid approach to diagnosis is unique in that
where H is the number of levels in the hierarchy, Fi are it uses the results of the structural isolation process to
cost functions for each level of the hierarchy and Ni is index cases. There is little overhead in retrieving rel-
the number of possible faults [19]. If the cost functions evant cases and matching is simplified since it is only
Fi are constant over all levels, then the cost of a non- applied to similar cases. We are considering CBR as
hierarchical diagnosis is F(N H ) while the cost of a a tool for assisting the operator in the final stage of a
hierarchical diagnosis is only (H × F(N )).2 However, diagnostic session. Once the structural isolation has
hierarchical decompositions of physical systems are produced a list of potential diagnoses, the operator can
sometimes too constraining and need to be relaxed at call on the CBR component of the system to validate
the lowest levels [23]. diagnoses or investigate other paths in the search tree.
Govindaraj and Su suggest that empirical constraints The CBR system is therefore called only after the struc-
should direct the formation of knowledge representa- tural isolation process.
tion in a format that reflects how experts solve prob- The remainder of this section describes the content of
lems [24]. They also observed that human diagnosis cases, the retrieval process and the matching algorithm
proceeds in a hierarchical manner, starting at higher currently being used to evaluate the case similarity.
levels of abstraction to generate hypotheses that guide
the diagnosis at lower levels. Hierarchical, structural 4.2.1. Case Representation. In general, a case stores
or conceptual data structures have already been found a fragment of a past experience. A case stores a past
useful for diagnosis applications (e.g., [22]). Slovic diagnostic scenario, consisting of a description of the
et al. pointed out that the effectiveness of decision fault that occurred (fault type, fault time, detecting
support systems for diagnosis depends on the ability of sensor), the series of pruning steps used to produce
the users to decompose the overall decision problem a list of potential diagnoses (i.e., the tests performed
into its constituent elements [23, 24]. during diagnosis and their values), the list of potential
While structural knowledge is usually available from diagnoses produced by the structural isolation process
engineering design documents and easily encoded, the and the correct diagnosis selected by the operator.
knowledge about failure modes and patterns tends to A case is completed once the operator has selected
be complex and of various types. In the case of sensor and confirmed a final diagnosis for the current diagnos-
based diagnosis, the queries to the database and their tic session. A diagnostic session where the structural
use in the sensor functions lead to numerous difficul- isolation process correctly produced the correct diag-
ties. These difficulties constitute the major differences nosis creates a “successful” case. A diagnostic session
between complex devices and electronic circuits where where the algorithm did not find the solution and where
failures and their consequences are straightforward to the operator had to give the correct diagnosis produces
characterize. a “failed” case. Both successful and failed cases con-
The models used by the ADMS methodology there- tain the fault information (i.e., the fault type, the fault
fore consist of a structural decomposition of the devices time and the sensor which detected the fault) and a flag
along with necessary conditions for substructures and indicating whether the case is a successful or a failed
basic components being potential diagnoses. They are case. Cases also contain the information generated
fault models in which all testing conditions use the real- throughout the structural isolation process. This infor-
time sensor data stored in the database through a set of mation is used to compare the current situation to a
predefined queries. past case. For each of the basic components that were
The output of the MBD diagnostic algorithm is a considered as potential diagnoses, cases also contain
ranked list of potential diagnoses from which the op- the list of its associated sensor functions and their val-
erator can (or not) select the final (supposedly correct) ues, triplets of the form: hsensor function, component,
diagnosis3 . The ranking involves relative levels of resulti. Finally, the list of diagnoses constructed by the
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

Combining Case-Based and Model-Based Reasoning 63

Figure 1. The case base and the structural decomposition.

diagnosis algorithm and the final diagnosis selected by components. Such components often share the same
the operator are included in a case. characteristics and are likely to appear in each other’s
lists of potential diagnoses. They will likely share the
4.2.2. Indexing and Storage of Cases. The index- same cases, or cases that are very similar to each other,
ing problem is the problem of efficiently retrieving the except for failed cases. This ensures a useful grouping
cases most relevant to the current situation; a com- of similar cases with “bridges” from one grouping to the
plete traversal of the case-base is usually too computa- next provided by failed cases. This indexing schema,
tionally expensive. A set of indices has to be designed based on the information generated by the struc-
to allow the system to select a subset of cases that will tural isolation process, is therefore both simple and
be analyzed further to try to solve the current prob- effective.
lem. As mentioned in Section 3, there is a trade-off
between efficiency and coverage. The more cases con- 4.2.3. Retrieval and Matching of Cases. The case-
sidered, the better the chances to retrieve the most rel- based reasoning system is a decision support system.
evant cases, but the less efficient the retrieval and the Cases are retrieved from the case base to evaluate and
matching processes are. criticize the current list of potential diagnoses (as ex-
The structural isolation process can be seen as a plained in Section 4.1). The operator can ask the system
rough estimate of the location of a component whose to explore its case base and to criticize or confirm a po-
failure explains the observed symptoms. It is reason- tential diagnosis or to suggest new diagnoses that were
able to use the list of potential diagnoses as a means of not generated by the structural isolation process. Re-
indexing the case base, leaving the values of the asso- trieved past successful cases help validate or criticize
ciated sensor functions for the matching step which is a the current potential diagnosis. If the current potential
finer judgement of similarity. For either a successful or diagnosis is supported by a previous successful case,
a failed case, we use each potential diagnosis produced the level of confidence in this potential diagnosis can
by the structural isolation process as an index for the be raised. If the correct diagnosis for the most sim-
case. Figure 1 illustrates this indexing schema. Each ilar case disagrees with all the suggested diagnoses,
case is stored at the bottom of the structural decomposi- and belongs to a failed case, the validity of the current
tion under the basic components it contains as potential diagnosis is lowered. The correct diagnosis stored in
diagnoses (we use pointers to cases to avoid duplication the failed case is extracted from the case base and pre-
of data structures). The dashed arrows originate from sented to the user as a new potential diagnosis that can,
failed cases and point to the components representing in turn, be evaluated and critiqued.
the correct diagnoses for those cases. Retrieved cases are matched with the current situa-
This indexing schema is satisfying because of tion, using finer criteria to evaluate similarity. Such cri-
the interdependencies that exist among “neighboring” teria include state information from both the past case
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

64 Féret and Glasgow

and the current situation. The measure of similarity and considering only the n best current potential
yielded by the matching algorithm is a normalized, diagnoses.
weighted sum of the number of sensor functions shar- Broadening the search for other potential diagnoses
ing the same value, the number of substructures shared uses failed cases. For example, if Comp1 (Fig. 1) is the
in the path followed during the session represented current best potential diagnosis, evaluating this diag-
by the past case and during the current session, the nosis will result in examining the failed Case3, which,
common characteristics of the symptoms, etc. All in turn, will bring the component Compn to the oper-
steps in the matching algorithm involve comparisons ator’s attention. If the operator decides to pursue this
of booleans or of reals and are therefore relatively path, the successful Case2, and the failed Case9 will
inexpensive. be retrieved.
Matching is implemented as a weighted sum of sim-
ilarities between the current situation and the past case.
5. Experimental Results
The relevant reasoning steps (traversing the decompo-
sition hierarchy) are compared as well as the results of
The goal of EAD is to suggest relevant alternative po-
the sensor functions applied to the basic components at
tential diagnoses without overwhelming the operator
the bottom of the hierarchy. Because of the indexing
with too many of them. To show that EAD reaches this
method described above, these components will serve
goal, we define 5 hypotheses, which, if demonstrated,
as pointers to cases that represent diagnostic sessions
will confirm that the goal is actually reached:
caused by similar or related failures. The matching
is effective because the knowledge contained in those
cases and on which the matching is based is relevant • The quality of the device models strongly influences
in both the current and the past cases. The match- the diagnostic performance;
ing algorithm is based on boolean values (the results • The CBR component of EAD should improve sub-
of the sensor functions), and is focused on the part stantially the diagnostic performance of a model-
of the system that is the most relevant to the current based diagnoser;
situation. • It should do so for a reasonable computational cost;
• It should remain accurate, i.e., not overwhelm the
user with potential diagnoses;
4.2.4. Evaluation and Critiquing of Cases. Evalua- • and therefore it should improve the overall level of
ting and critiquing is an iterative process. The op- expertise of the system.
erator can discard potential diagnoses, generate new
ones, evaluate and criticize them, etc. There are two Figure 2 summarizes the relationship between the five
approaches to broadening the search for new poten- experiments run to provide evidence for the hypotheses
tial diagnoses. The first way explores the ranked list listed above.
of potential diagnoses from the highest to the lowest Testing the effectiveness of EAD requires an initial
ranked diagnoses. The operator can choose to evaluate case base of diagnostic cases. Such a case base could
the first potential diagnosis as the initial iteration, then be created with the normal running of the system. For
the second diagnosis (with a lower confidence level) at testing purposes, however, a simulator was constructed
the second iteration etc. For example, if a diagnostic for the Fairing Servicing Subsystem (FSS), a robotic
session produces Comp1 and Compk as potential diag- device, to automatically generate sensor data and the
noses, the operator may choose to evaluate Comp1 first. corresponding sensor function boolean values, repre-
The second way is to lower the matching threshold be- senting single faults. This simulator was used to gen-
tween the current situation and a past case. If the first erate initial case bases and to simulate test cases for
iteration does not yield any satisfying new potential di- evaluating the CBR component of the system.
agnosis, the operator may choose to lower the matching The hybrid approach described above is imple-
threshold until a matching case is found. For example, mented as an interactive process where the human op-
in Fig. 1, a high matching threshold might only ex- erator explores the case base indexed by the structural
tract the successful Case1 from the case base. A lower decomposition of the device. To test our approach,
threshold, set in the next iteration, would retrieve the we have implemented a non-interactive version of the
failed Case3 as well. These two ways can be combined same program that retrieves all the cases stored un-
by setting a lowest admissible matching threshold der the potential diagnoses produced by the structural
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

Combining Case-Based and Model-Based Reasoning 65

Figure 2. Characteristics of the experiments.

isolation. We progressively degrade the model, start- the phenomenon that EAD is hopefully compensating
ing from a model that produces near-perfect diagnostic for. In addition, we designed four experiments (num-
performance, and moving towards models that contain bered 2 to 5) that exhibit different properties with re-
errors. This degradation process takes place in the sen- spect to the abilities of the CBR component of EAD
sor functions, which are randomly selected and failed. to retrieve, match and adapt cases. Experiments 2 to
A flawed sensor function returns an incorrect result, 4 degrade the model by failing sensor functions. Ex-
falsely describing the state of the device. The model periment 5 degrades the model by altering the com-
for the Fairing Servicing Subsystem contains 19 sensor binations of sensor functions in failure mechanism
functions, describing 15 failure patterns (conjunctions patterns so that they “fire” under incorrect condi-
of sensor functions) for 100 components of 8 different tions. The goal of Experiment 5 is to show that other
component types, e.g., motors (4 failure patterns), ca- types of errors in the model are corrected equally well
bles (2 failure patterns), or fuses (2 failure patterns). by the CBR component of the system. The char-
The following experiments involved running 77 simu- acteristics of these experiments are summarized in
lations of different faults occurring in two modules of Fig. 2.
the Fairing Servicing Subsystem containing 36 com- Other errors can also be simulated and used for an
ponents and 7 sensors. experimental evaluation of our approach. We did in-
Some simplifying assumptions were made in carry- clude errors in the structural knowledge such as mis-
ing out the following experiments. First, we assumed placed or missing components. The experimental re-
that all possible failures for all components are equally sults showed that the CBR system quickly focuses
probable. Although this does not reflect reality, it does on such errors by producing groups of failed cases
present—to some degree—a worst case scenario. Sec- with similar characteristics. Our experience with real-
ondly, we assume that the sensor information is ac- world devices also showed that engineers could eas-
curate, i.e., the sensors are not included among the ily locate and fix this type of errors in the device
components that may fail. Redundant sensors could models.
be put on the device as well as extra sensor functions Two different retrieval mechanisms are used.
checking on the consistency of measurements from re- Experiment 2 retrieves all cases stored under the com-
dundant sensors to consider sensors as potential fault ponents that are possible components for the current
sources4 . diagnostic session. Experiments 3 to 5 retrieve all
cases that are stored under all components of the same
component types as the possible components for the
5.1. Experimental Method current diagnostic session. This latter retrieval mech-
anism calls for an adaptation mechanism that is de-
Five experiments were carried out to test the hypotheses scribed later.
listed above. Experiment 1 aims at showing the effect Experiment 2 uses the similarity measure algorithm
of model degradation on diagnostic performance, i.e., described in Appendix 1 (Algorithm 1). This algorithm
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

66 Féret and Glasgow

compares the fault information, the lists of possible For each experiment, we measure the failure rate,
modules, streams and components. It then compares the number of retrieved cases, the number of matched
the information derived from the structural isolation cases, the number of potential diagnoses produced by
process in both the current situation (CC) and the past the system (with the CBR system contributing some
case (PC). Sensor function values are compared over diagnoses), and a competence value5 . These measure-
the set of potential diagnoses for CC. Potential diag- ments are used next to show the validity and effective-
noses are possible components for which at least one ness of EAD.
failure mechanism has fired. The following sections show the diagnostic perfor-
Experiments 3 to 5 use a similarity measure based mance decrease relative to model degradation, the evo-
exclusively on the last part of the structural isolation lution of the system’s failure rate with the accumulation
process. The algorithm is described in Appendix 2 of experience, the computational cost of CBR relative
(Algorithm 2). There is no comparison of the informa- to the case base size and the evolution of the accuracy
tion derived from the rule-out phase of the diagnostic and overall competence of the system relative to accu-
algorithm except for its results: the list of possible mulation of experience.
components, which is compared in a similar fashion as
in the algorithm described above. The last step of the
algorithm is different in two ways: (1) the algorithm 5.2. Performance Degradation
compares sensor function values for components of the from Model Degradation
same types and not only for the same components, (2)
all possible components of CC are considered, as op- If the diagnostic performance was not influenced by
posed to only the potential diagnoses. The first dif- the quality of the device model, the CBR component
ference is a consequence of the implicit generalization of EAD would be of little value. We therefore have to
of the cases. The second gives a broader base for the show that the quality of the device model does influence
comparison and seems to yield better results. the diagnostic performance. This is in itself not sur-
The similarity measures are important to the experi- prising. Nevertheless, it is useful to have a numerical
ments. Some small-scale measurements were done on estimate of this influence.
the similarity measures to estimate the adequate thresh-
old for similarity. A few flawed sensor function combi- 5.2.1. Hypothesis. The hypothesis of this experi-
nations, and a few pairs of test cases were used to mea- ment is that the diagnostic performance of the system
sure their similarity. The pairs of cases were classified decreases with the quality of the device model on which
as similar or dissimilar. Figure 3 shows the means, and the structural isolation process is based (Hypothesis 1).
standard deviations for these 2 classes for both algo-
rithms. For Algorithm 1, similarity is used to recognize
the identical cases. The two classes of pairs of cases 5.2.2. Method. The 77 simulations are run with an
(dissimilar and similar) are not well separated by the increasing number of flawed sensor functions (from 0
algorithm. This does not matter since the purpose of to 15). The results of these simulations are compared to
the algorithm is to recognize the pairs that are identical. the simulated fault. If the correct diagnosis (i.e., the in-
The threshold should therefore be set high compared put of the simulator—a component and a failure mode)
to the mean of the “class” of similar cases. Experi- is not present in the set of potential diagnoses, the di-
ment 1 uses Algorithm 1 with a threshold of 8.6, which agnostic case is considered as a failed case, otherwise
is slightly lower than the measure produced by the al- it is a successful one.
gorithm for identical cases (see Fig. 3). Algorithm 2
is used in a context where generalization occurs. It is 5.2.3. Results. Figure 4 shows that the diagnostic
important that the two classes of cases be clearly sepa- performance degradation is approximately linear in the
rated. Experiment 3 uses Algorithm 2 with a threshold number of flawed sensor functions. The linearity of
of 6.65, i.e., the higher end of the measures produced the degradation in Fig. 4 is the result of a balanced
for pairs of dissimilar cases. Experiments 4 and 5 use use of sensor functions in the failure patterns. Fail-
the same algorithm with a threshold of 7.7, i.e., the ure patterns consist of 3 to 6 sensor functions, possibly
mean of the values produced for pairs of similar cases negated. Intuitively, Fig. 4 corresponds to the con-
minus the standard deviation (for measures for similar cept that the worse the model is, the less accurate the
cases). diagnostic process is.
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

Combining Case-Based and Model-Based Reasoning 67

Figure 3. Similarity measures’ distributions.

5.3. Diagnostic Performance Improvement The goal of the second set of experiments is to mea-
sure the improvement of diagnostic performance in re-
The previous experiment suggested that the qual- lation to the accumulation of experience for the var-
ity of the device model strongly influences the ious experimental setups. The failure rate, i.e., the pr-
diagnostic performance measured with the failure oportion of simulations for which the system fails to
rate of the system. We now want to show that, the produce the correct diagnosis, is used for these exper-
CBR component of EAD can indeed bring the fail- iments. This experiment is run without the CBR sys-
ure rate down with the accumulation of experi- tem to stress the problems caused by imperfect device
ence. models.
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

68 Féret and Glasgow

two experiments do not reach perfect diagnostic perfor-


mance because the adaptation mechanism is not perfect
and fails to adapt adequately some matched cases to the
current situation.
Figure 5(d) shows that the CBR component also im-
proves performance for a different type of error. The
learning curve is even steeper than for Figs. 5(b) and (c)
since the induced errors in Experiment 5 affect more
diagnostic situations than in Experiments 3 and 4 and
therefore can be corrected faster.
These results show that EAD does improve diagnos-
tic performance as the amount of experience (number
of cases) increases. It also shows that the ability to
Figure 4. Experiment 1: Performance degradation. retrieve different, but relevant cases, and to adapt them
to the current diagnostic situation, described through
the tests done by the structural isolation process, is the
5.3.1. Hypothesis. The hypothesis of this experi- key to faster learning.
ment is that the failure rate of the system decreases
with the amount of accumulated experience (Hypoth-
esis 2). 5.4. Computational Cost of EAD

5.3.2. Method. In this experiment, case bases of diff- The previous experiments indicated that EAD can be
erent sizes are used for different numbers of flawed sen- useful in compensating for errors in the device model.
sor functions. For each combination of flawed sensor The more experience is accumulated, the more EAD
functions, and for each case base size, the 77 simula- can compensate. Another part of our initial hypothesis
tions are run on 10 randomly constructed different case is that the computational cost of EAD remains limited.
bases. The set of training simulations does not inter-
sect with the set of testing simulations (as usually done 5.4.1. Hypothesis. The hypothesis of this experi-
in machine learning experiments). Both sets contain ment is that the computational cost of the CBR sys-
simulations producing both success and failed cases. tem remains within reasonable bounds, as experience
In Experiment 2, the case-based component is very is accumulated and used (Hypothesis 3).
simple: the retrieval mechanisms retrieves all cases
stored under the components that are potential diag- 5.4.2. Method. The setup for the four experiments
noses produced by the structural isolation process. is the same as described in Section 5.1. The number
The matching Algorithm 1 is used with the similarity of retrieved cases is used to measure the computational
threshold set to 8.6. There is no adaptation mechanism, cost of the CBR component. The CBR system applies
i.e., the final diagnosis stored in a matched case is given the matching algorithm to these cases. Only the cases
to the user “as is”. Experiments 3 to 5 retrieve all cases producing a high enough similarity measures are con-
that are stored under components of the same type as sidered for adaptation. Figures 6(a)–(d) show the aver-
the potential diagnoses produced by the structural iso- age number of retrieved diagnoses. This measure is a
lation process and that belong to the same module. The reliable measure for the computational cost of the CBR
similarity threshold for Algorithm 2 is 6.65 for Experi- component: the computational cost of the retrieval is
ment 3 and 7.7 for Experiments 4 and 5. proportional to this number. The cost of matching and
of adaptation are constant for each retrieved case, and is
5.3.3. Results. Figure 5(a) shows that even a simpli- therefore proportional to the number of retrieved cases.
stic case-based reasoner, with ability to adapt its cases
can be useful. Figures 5(a)–(c) show that better re- 5.4.3. Results. Figure 6 shows the number of retri-
sults are achieved with case-based reasoners able to eved cases. Only some of these cases will be judged
perform case adaptation. Figure 5(c) shows a very similar enough to the current situation and be given to
similar learning curve as in Fig. 5(b), even though the the users as potential diagnoses (we call these last cases
similarity threshold is higher in Experiment 4. These “matched” cases).
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

Combining Case-Based and Model-Based Reasoning 69

(a) (b)

(c) (d)

Figure 5. Failure rates relative to casebase size: (a) Experiment 2, (b) Experiment 3, (c) Experiment 4, (d) Experiment 5.

Figure 6(a) shows that the computational cost of order to achieve gains in performance. In other words,
EAD, used as a simple retrieval mechanism increases these results show that EAD can be scaled up to larger
linearly with the size of the case base. The curves re- devices or larger case bases.
main reasonably close to flat for moderately degraded
devices models (because there are few cases to be re-
trieved), indicating that even a simple EAD schema 5.5. Accuracy of EAD
would be useful for device models approaching per-
fection. Figures 6(b)–(d) show that a more selective The previous experiments suggest that EAD is effective
retrieval mechanism is possible whose cost flattens out at improving the success rate of the system for a com-
when the case base grows larger. Combined with re- putational cost that can remain reasonable. Another
sults from Figs. 5(b)–(d), these results clearly demon- aspect of our global hypothesis is trying to prevent the
strate that some significant gains in diagnostic perfor- operator from being overwhelmed by potential diag-
mance can be achieved for a cost than remains reason- noses suggested by the system. We want to show that
able. Note that there is no difference between Figs. 6(b) the system can stay reasonably focused and accurate,
and (c) since the difference in the similarity threshold regardless of the accumulated experience.
should influence the number of matched cases, not the
number of retrieved cases. 5.5.1. Hypothesis. The hypothesis of this set of resul-
These results show that EAD is applicable to larger ts is that the number of suggestions, i.e., potential di-
devices or to larger case bases. Its costs can be kept agnoses, presented to the operator stays reasonable as
under a certain limit while the case bases grow larger in experience is accumulated and re-used (Hypothesis 4).
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

70 Féret and Glasgow

(a) (b)

(c) (d)

Figure 6. Number of retrieved cases: (a) Experiment 2, (b) Experiment 3, (c) Experiment 4, (d) Experiment 5.

5.5.2. Method. The setup for the four experiments (the same experiment with a higher similarity thresh-
is the same as described in Section 5.1. Figures old) and Experiment 5 show a constant increase up to
7(a)–(d) show the average number of potential diag- a threshold where the number of potential diagnoses
noses produced by the EAD method. A potential diag- remains constant.
nosis can be produced by the device model itself or be These results suggest that the importance of the
the result of a matched case. The influence of the CBR similarity thresholds in limiting the number of poten-
component is indirectly read by observing the increase tial diagnoses produced by the system. An adequate
of potential diagnoses produced by the system. With- threshold leads to results showing that EAD could be
out the CBR system, the number of potential diagnoses scaled-up without deterioration of accuracy.
produced by the system would stay constant relative to
the case base size (note that there is a small increase in 5.6. Overall Competence of the System
that number relative to the quality of the model).
The previous experiments have already indicated that
5.5.3. Results. Figures 7(a)–(d) show the average EAD does improve the diagnostic performance of the
number of potential diagnoses for the four experiments. system, does so for a computational cost that can re-
The increase is proportional to the case base size for main reasonable, and without producing too many po-
Experiment 2, but the multiplicative constant is small tential diagnoses, so that the results can actually be
and the overall number of potential diagnoses produced used by a human operator. This set of results aims at
by the system remains low. Experiment 3 shows high summarizing all these results into one global measure
numbers of potential diagnoses, while Experiment 4 of competence.
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

Combining Case-Based and Model-Based Reasoning 71

(a) (b)

(c) (d)

Figure 7. Number of potential diagnoses: (a) Experiment 2, (b) Experiment 3, (c) Experiment 4, (d) Experiment 5.

The goal of this set of results is to show that the over- (enough) result, given the complexity of the device,
all competence of the system does improve over expe- and the sensor density. This measure of competence is
rience. The goal of the CBR component of the system only interesting as a relative value. Note that a perfect
is to suggest diagnoses that might have been forgotten diagnoser, one that outputs only one correct diagnosis
by the model-based method. The effectiveness of the for all possible sets of symptoms would have a com-
CBR component is directly related to the number of petence of 1. Competence increases with the average
diagnoses it suggests: the more irrelevant diagnoses success rate, and decreases with the average number of
it produces, the less “expert” its performance should potential diagnoses produced by the whole diagnostic
be considered. One possible measure for the overall process.
competence of the diagnostic system. This measure is
defined as follows:
5.6.1. Hypothesis. The hypothesis is that the ove-
rall competence of the system does increase as ex-
Competence perience is acquired and re-used for new diagnostic
1 − average failure rate situations (Hypothesis 5).
=K×
average number of potential diagnoses
where K represents an integer constant accounting for 5.6.2. Method. The experimental setup for the experi-
the average number of potential diagnoses a compe- ments is the same as described in Section 5.1. The
tent diagnostic system would output. For example, competence measure showed above is used for this set
for the FSS, a list of 4 potential diagnoses is a good of experiments. This measure takes into account both
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

72 Féret and Glasgow

(a) (b)

(c) (d)

Figure 8. Competence: (a) Experiment 2, (b) Experiment 3, (c) Experiment 4, (d) Experiment 5.

the failure rate and the number of potential diagnoses The competence measures for Experiments 3 to 5
produced by the system. stop increasing when the case base contains cases that
can be generalized to cover all diagnostic errors result-
5.6.3. Results. Figure 8(a) shows that the overall ing from the device model’s inadequacies.
competence increases linearly with the case base size in The results for Experiments 3 and 4 is a compe-
a setting where CBR system performs mere retrieval. tence measure that decreases with the size of the case
This is due to the fact that there is no implicit gen- base for device models than are almost perfect (with a
eralization and no adaptation performed by the CBR small number of flawed sensor functions). This is due
system. Each stored cases is used “as is”. This is to the similarity threshold set at the upper end of the
a very “safe” way of improving the system’s compe- class of dissimilar cases. As a result, the retrieval and
tence since the number of potential diagnoses remains matching processes of the CBR system are not accurate
low. However, the learning curve is very smooth. enough and match too many cases, including numerous
Figures 8(b) and (c) show steeper increases in the cases that should not have been retrieved. The decrease
competence measure. Experiment 4 yields better re- in competence is only marginal for small numbers of
sults since the CBR system improves the discrimination flawed sensor functions and is mostly stable for higher
of relevant cases from irrelevant ones. Experiment 5 values.
shows results lower than those of Experiment 4 for bet- These results are encouraging. They show improved
ter models and higher than those of Experiments 4 for overall performance in all situations, even for a sim-
more degraded device models. This is due to the type plistic CBR system. However, Experiments 4 and 5
of errors induced in the two experiments. show that the competence of a diagnoser starting with
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

Combining Case-Based and Model-Based Reasoning 73

a degraded model cannot reach the competence of the to recall only relevant cases to limit the computational
same diagnoser starting with a better model and accu- cost of the CBR component and to positively match
mulating experience at the same rate. This calls for ma- the most relevant ones to improve the accuracy of the
chine learning techniques that would update the model CBR system. The key to the former is an adequate
itself instead of storing past experiences and reusing retrieval mechanism. The key to addressing the lat-
them to “patch up” the results of the model-based di- ter is the similarity measure. The similarity measure
agnosis process. The type of learning that would occur for Algorithm 2 is not particularly good. It does not
would depend on the model. For example, trying to discriminate well between pairs of similar cases and
modify the model of the FSS would be extremely diffi- dissimilar cases. However, with an adequate similarity
cult because of the sensor functions. On models using threshold, the CBR system does improve the perfor-
real-time sensor data, conceptual clustering methods mance of the system significantly.
could be used on the sensor data itself, or on a more These experiments are run under the unrealistic and
compact representation of it to decide in which cases unfavorable assumption that no fault occurs twice. This
sensor functions could be trusted. represents a worst possible case for the CBR system,
Recall that these experiments use only part of the since there will never be a case identical to the current
model. If the entire model was used (representing app- situation. Also, the CBR system only uses failed cases
roximately 4 times more components and possible to learn. Therefore, the case base only contained failed
single fault situations), the graphs in Figs. 8(b)–(d) cases. An interesting follow-up would be to try to
would be the same, showing a diagnostic system reach- isolate the parts of the device model that contain errors.
ing its peak competence after only a few cases. The This can be done by exploiting the model to concentrate
graph of Fig. 8(a) would show a continuous increase in the simulations on the same parts of the model that
competence. were used to explain a fault that resulted in a failed
case. In the case of the FSS, statistics could be kept
on the sensor functions that should have been used to
5.7. Closing Remarks on the Experiments explain a missing diagnosis. This is because only the
failed cases are used for learning.
Experiment 1 provides a reference point for the perfor- Better learning performance could be achieved if
mance degradation as a function of the device model successful cases were used as well. For example, they
degradation. Experiments 2 to 5 show that: (1) even could be used to improve the ranking of the potential
a simplistic CBR system can be useful, (2) appropri- diagnoses, that is usually based on engineering data
ate retrieval, matching and adaptation mechanisms can about meantime between failure, expected life-time,
lead to substantial performance improvements, (3) the frequency of failures etc. A finer retrieval mechanism
similarity threshold for the matching algorithm is im- could be derived from this information unavailable for
portant in the performance improvement, (4) the cost of these experiments. For example, only the first N poten-
the CBR component can remain low, and (5) EAD can tial diagnoses of a diagnostic situation could be used as
handle different types of errors in the device models. indices for the case representing this situation. We can
The experiments described above show that it is pos- only suggest that this would result in fewer retrieved
sible to use CBR to effectively exploit a case base of cases and fewer matched cases, giving higher com-
past experiences to improve the performance of an au- petence levels, especially for high numbers of flawed
tomated diagnostic system. This accounts for possi- sensor functions.
ble errors in the device model used by the diagnostic In conclusion, the experiments show that, even with
system. partial information for the CBR system, not allow-
The experiments show that the indexing mechanism ing the ranking of potential diagnoses, and under a
is effective. All relevant failed cases are retrieved con- worst possible probability distribution for the faults,
sistently. It also shows that the method works even the method still works effectively.
with a very simple CBR system with no adaptation This approach could also be tested for erroneous
mechanism. They also show that a simple retrieval structural knowledge. Our experience showed that
method, with a simple similarity measure, allows the these types of errors are typically easier to detect than
CBR system to recall past failed case experiences and errors in sensor functions. We did include (sometimes
adapt them to the current situation. The challenge is unintentionally) errors in the structural knowledge
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

74 Féret and Glasgow

such as misplaced or missing components. The the indexing schema provided by the other problem-
experimental results showed that the CBR system very solving algorithm. Cases are indexed by relevant pi-
quickly focuses on such errors by producing groups eces of knowledge, organized in a hierarchical man-
of failed cases with similar characteristics. Our expe- ner. The other problem-solving algorithm (be it rules
rience with real-world devices also showed that engi- triggering or a structural isolation process) can be seen
neers could easily locate and fix this type of errors in as a indexing schema that extensively uses background
the device models. knowledge. Both systems can therefore be seen as
We can therefore claim that the results showed in this constrained instances of explanation-based indexing
papers were obtained in unfavorable statistical circum- [3].
stances (all possible device model errors are equally The CBR component of the EAD is generic. It does
probable and so were all failure modes) and for device not depend on the device or on which type of model
model errors which were the most difficult for human is used. The structural decomposition is device depen-
operators to detect, locate and fix. dent. Its completion or correction by the CBR com-
ponent is not. In fact, EAD can also be incorporated
in other model-based approaches to diagnosis, includ-
6. Discussion ing first-principle approaches based on correct behav-
ior device models. This is a definite advantage, both
This paper presents EAD, a hybrid approach to diag- in the framework of the genericity of the ADMS as a
nosis using experience to aid in the search for potential diagnostic system and for the applicability of CBR as
diagnoses. The example of EAD showed in this paper a complement to other problem-solving techniques.
is based on a structural isolation search for potential di- The experimental results, given for the device model
agnoses, enhanced by contributions from an integrated errors that were most difficult to detect, locate and fix,
CBR component that assists the human operators in also show that the EAD method can be scaled-up to
their final decision by using both successful and failed larger devices if an adequate similarity measures can
case experiences. Such assistance is useful because the be found, and an appropriate similarity threshold can
structural decomposition and the pruning rules associ- be determined ahead of time. Under these conditions,
ated with it are not guaranteed to be either consistent the computational cost of the CBR system remains rea-
or complete and because human operators might not sonable, especially compared to the potential cost of an
consider all the cues that are available to them. NP-hard abductive diagnostic system.
Compared to an all-model-based approach to diag- Cases could also be used effectively in diagnosing
nosis, EAD addresses a number of problems. CBR multiple faults. We have not tested this aspect of our
allows the system to improve on the model built by approach yet. If a fault is confirmed by a case X , an-
humans. It overrules some mistakes that can be made other case Y could also be retrieved that has case X ’s
in the design and implementation of this model. This final diagnosis as one of its potential, or final diagnoses.
is done in a computationally inexpensive way, by us- The final diagnosis of Y could be examined as a po-
ing the decomposition tree as the basis for indexing tential diagnosis for the current situation, potentially
previous cases. This original indexing method allows uncovering a double fault situation. Multiple faults
for an accurate and effective retrieval of relevant previ- could therefore be diagnosed using the same hybrid
ous cases and avoids the problems of computational approach with no added complexity.
complexity encountered by other hybrid CBR sys- As discussed in Section 5.7, the results showed in
tems (e.g., [34, 39]) in their retrieval and matching this paper are for errors in device models that proved
tasks. to be the most difficult for human operators to detect,
Experimental results show that the performance gain locate and repair. They also turned out to be the most
brought by the CBR system is significant. The CBR difficult errors for the CBR system to recover from. We
system improves the failure rate of partially incorrect therefore believe that the set of experiments presented
models without overwhelming the operator with po- here constitutes a convincing sample. A possible lim-
tential diagnoses. These results suggest that consid- itation of our approach resides in the nature of the de-
ering CBR as a way to improve an existing search vices. Devices that are not hierarchical or present a
method is a valid approach. This solidifies the results wide variety of very different components very loosely
presented in [21]. The power of Golding and Rosen- connected to each other may not be as amenable to our
bloom’s method as well as ours comes mostly from approach as the FSS or the RBVS.
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

Combining Case-Based and Model-Based Reasoning 75

The experimental results show the source of the SM = similarity measure.


EAD approach’s power. CBR postulates that simi- SMi = partial similarity measures, initialized to 0.
larity is a reasonable estimate for relevance. While
this is not always the case with “pure” CBR systems, if the detecting sensors for PC and CC are the same,
it becomes true more often with hybrid systems. In then SM1 := 1
the EAD approach, a given error in the device model else if the detecting sensors are of the same
tends to cause related diagnostic errors which result in sensor type, then SM1 := 0.5 endif.
cases with similar characteristics. Therefore, in this endif.
situation the assumption that similarity accounts for if the fault types of CC and PC are the same, then
relevance holds. The EAD approach will not work SM1 := SM 1 + 1 endif.
very well on devices with characteristics such that
this assumption holds rarely. A characterization of for each possibly faulty module pm of CC,
such devices constitute an interesting topic for future if pm is a possibly faulty module for PC,
research. then SM2 := SM 2 + 1 endif.
It is clear that the EAD approach is applicable to a endfor.
whole range of searching and problem-solving meth- Divide SM2 by the number of possibly faulty
ods. An interesting extension to our work would be modules for CC.
to apply this hybrid approach to other domains
such as natural language parsing and understand- for each possibly faulty stream ps of CC,
ing, planning or game playing, where hierarchies if ps is a possibly faulty stream for PC,
and context information expressed by context func- then SM3 := SM 3 + 1 endif.
tions, equivalent to the sensor functions, are readily endfor.
available. Divide SM3 by the number of possibly faulty
From a knowledge-base validation and verification streams for CC.
point of view, this method is of interest when there
exists a simulator for the monitored device. Simple for each possibly faulty component pco of CC,
models and simulators are common for engineering if pco is a possibly faulty component for PC,
devices. A rough model could be used by an automated or if there is a possibly faulty component
simulation procedure as a starting point from which for PC of the component type as pco
a better model could be derived automatically, using then SM4 := SM 4 + 1 endif.
CBR for refining the initial model. endfor.
This paper presents how a hybrid model-based/case- Divide SM4 by the number of possibly faulty
based methodology permits the relaxation of the com- components for CC.
pleteness and consistency constraints imposed by the
model-based diagnosis approach, and helps overcome for each potential diagnosis’ component pd for CC
shortcomings in human capabilities. We show how if pd is also in PC’s list of potential diagnoses,
CBR can guide the human operator in the last phase then for each sensor function pd sf for pd
of the diagnostic process, using previous experiences in CC
indexed by the state of the device. The paper also con- if pd sf has the same boolean value
tributes to the area of CBR, by showing that it is well in CC and in PC,
suited to applications where it is combined with an al- then SM5 := SM 5 + 1 endif;
ready existing, but imperfect, method or paradigm for endfor.
problem-solving. endif.
endfor.
Divide SM5 by the number of pd sf shared by CC
Appendix I and PC for all pds.
Similarity Measure for Experiments 2 and 3: W1 := 1; W2 := 1; W3 := 1; W4 := 2; W5 := 3;
Algorithm 1 X
i=5
SM := SM i × Wi .
CC = current case; PC = past case; i=1
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

76 Féret and Glasgow

Appendix 2 Notes

Similarity Measure for Experiments 4 and 5: 1. The ADMS, as well as the models of the two robotic devices used
as tests were developed in collaboration with Spectrum Engineer-
Algorithm 2
ing Corp. Ltd., Peterborough, Ontario, under a contract with the
Canadian Space Agency (STEAR program).
CC: current case; PC: past case; 2. The function F is rarely constant, but grows with the abstraction
SM: similarity measure. levels. The more abstract a level, the more complex its behaviors
SMi , 4 ≤ i ≤ 5: partial similarity measures, and the more costly the diagnostic cost. However, the behav-
ioral complexity rarely grows as fast as the structural complexity
initialized to 0. decreases.
3. This paper does not address how the operator chooses the final
for each possibly faulty component pco of CC, “correct” diagnosis. The reasoning and validation process used
if pco is a possibly faulty component for PC, by the operator is outside the scope of this paper. We assume that
visual inspection, testing of the repairs etc., allows the operator to
or if there is a possibly faulty component for eventually assess, with a very high degree of confidence, which
PC of the component types as pco component failed, and input it into the system.
then SM4 := SM 4 + 1 endif. 4. The model implemented for the RBVS accounts for the possibility
endfor. of failing sensors. The current diagnostic algorithm has been
tested successfully on this model. The effectiveness of EAD
Divide SM4 by the number of possibly faulty in multiple fault situations is currently been investigated on this
components for CC. model.
5. Recall and precision measures derived from Information Retrieval
for each possibly faulty component pco for CC were also extracted. They provided insights on the details of the
working of the CBR system.
if pco is in PC’s list of potential diagnoses,
or if there is a component of the same type
as pco that is a potential diagnosis for PC
then for each sensor function pco sf References
for pco in CC
if pco sf has the same value in 1. K.D. Ashley and E.L. Rissland, “Compare and contrast: A test
of expertise,” in Proceedings of Fifth National Conference on
CC and in PC, then Artificial Intelligence, Morgan-Kaufman, 1987.
SM5 := SM 5 + 1 2. E.R. Bareiss and B.W. Porter, “Protos: An examplar-based
endif; learning apprentice,” in Proceedings of the Fourth International
endfor. Workshop on Machine Learning, June 1987, pp. 12–23.
3. R. Barletta and W. Mark, “Explanation-based indexing of cases,”
endif. in Proceedings of the Sixth National Conference on Artificial
endfor. Intelligence, 1988, pp. 541–546.
Divide SM5 by the number of pco sf 4. E. Blevis, R. Burke, J.I. Glasgow, and N. Duncan, “The life
shared by CC and PC for all pcos. analysis and depreciation integrated exmplar system (ladies),”
International Journal of Expert Systems, Special Issue on Case-
Based Reasoning, vol. 4, no. 2, pp. 141–156, 1992.
W4 := 1; W5 := 7; 5. T. Bylander, D. Allemang, M.C. Tanner, and J.R. Josephson,
X
i=5 “The computational complexity of abduction,” Artificial Intelli-
SM := SM i × Wi . gence, vol. 49, pp. 25–60, 1991.
i=4 6. A.M. Collins and M.P. Quillian, “Retrieval time from seman-
tic memory,” Journal of Verbal Learning and Verbal Behavior,
vol. 8, pp. 240–247, 1969.
7. L. Console, L. Portinale, and D.T. Dupre, “Focusing abductive
Acknowledgments
diagnosis,” AI Communications, vol. 4, nos. 2/3, pp. 88–97,
1991.
This research was supported through a contract from 8. L. Console, D.T. Dupre, and P. Torasso, “A theory of diagno-
the Canadian Space Agency (STEAR program), and sis for incomplete causal models,” in Proceedings of Eleventh
a scholarship and an operating grant from the Natural IJCAI, Detroit, 1989, pp. 1311–1317.
9. J. de Kleer, “Diagnosis with behavioral modes,” in Proceedings
Sciences and Engineering Research Council (NSERC)
of the Eleventh International Joint Conference on Artificial In-
of Canada. We also would like to thank Spectrum telligence, Detroit, 1989, pp. 1324–1330.
Engineering Corporation Ltd., Peterborough, Ontario, 10. J. de Kleer, “Using crude probability estimates to guide diagno-
Canada. sis,” Artificial Intelligence, vol. 45, no. 3, pp. 381–391, 1990.
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

Combining Case-Based and Model-Based Reasoning 77

11. J. de Kleer, “Optimizing focusing model-based diagnosis,” in Case-Based Reasoning, edited by K. Hammond, 1989, vol. 2,
Proceedings of the Third International Workshop on Principles pp. 115–118.
of Diagnosis, Rosario, Washington, 1992, pp. 26–29. 28. J.L. Kolodner, “Maintaining organization in a dynamic long-
12. J. de Kleer and B.C. Williams, “Diagnosis multiple faults,” Ar- term memory,” Cognitive Science, vol. 7, pp. 243–280, 1983.
tificial Intelligence, vol. 32, pp. 97–129, 1987. 29. J.L. Kolodner, Retrieval and Organizational Strategies in Con-
13. M.P. Féret and J.I. Glasgow, “Generic diagnosis for mechanical ceptual Memory: A Computer Model, Lawrence Erlbaum, 1984.
devices,” in Proceedings of the Sixth International Conference 30. J.L. Kolodner, “Capitalizing on failure through case-based in-
on Applications of Artificial Intelligence in Engineering, Oxford, ference,” in Proceedings of the Ninth Annual Conference of the
UK, Computational Mechanics Publications, Elsevier Applied Cognitive Science Society, Lawrence Erlbaum: Hillsdale, NJ.
Science, July 1991, pp. 753–768. 1987, pp. 155–162.
14. M.P. Féret and J.I. Glasgow, “Case-based reasoning in model- 31. J.L. Kolodner, “Improving human decision making through
based diagnosis,” in Proceedings of the Seventh International case-based decision aiding,” AI Magazine, vol. 12, no. 2, pp. 52–
Conference on Applications of Artificial Intelligence in En- 68, 1991.
gineering, Waterloo, Canada, Computational Mechanics Pub- 32. J.L. Kolodner and R.M. Kolodner, “Using experience in clinical
lications, Elsevier Applied Science, July 1992, pp. 679– problem solving: Introduction and framework,” IEEE Transac-
692. tions on Systems, Man, and Cybernetics, vol-SMC. 17, no. 3,
15. M.P. Féret and J.I. Glasgow, “Hybrid case-based reasoning for pp. 420–431, 1987.
the diagnosis of complex devices,” in Proceedings of AAAI-93, 33. J.L. Kolodner and R.L. Simpson, “The mediator: Analysis of
Washington, D.C., July 1993, pp. 168–175. an early case-based problem solver,” Cognitive Science, vol. 13,
16. M.P. Féret and J.I. Glasgow, “Explanation-aided diagnosis for no. 4, pp. 507–549, 1989.
complex devices,” in Proceedings of the Twelveth National 34. P. Koton, “Reasoning about evidence in causal explanations,” in
Conference on Artificial Intelligence (AAAI-94), Seattle, USA., Proceedings of AAAI-88, 1988, pp. 256–261.
Aug. 1994. 35. T.P. Martin, J.I. Glasgow, M.P. Féret, and T.G. Kelley, “A
17. M.P. Féret, J.I. Glasgow, D. Lawson, and M.A. Jenkins, “An ar- knowledge-based system for fault diagnosis in real-time engi-
chitecture for real-time diagnosis systems,” in Proceedings of the neering applications,” in Proceedings of DEXA’91-International
Third International Conference on Industrial and Engineering Conference on Database and Expert System Applications,
Applications and Expert Systems, Charleston, SC, July, 1990, Berlin, Germany, Aug. 1991, pp. 287–292.
pp. 9–15. 36. R.W. Milne, “Strategies for diagnosis,” IEEE Transactions on
18. G. Friedrich, “Theory diagnoses: A concise characterization of Systems, Man, and Cybernetics, vol. SMC-17, no. 3, pp. 333–
faulty systems,” in Proceedings of the Third International Work- 339, May/June, 1987.
shop on Principles of Diagnosis, Rosario, Washington, 1992, 37. I. Mozetic, “Reduction of diagnostic complexity through model
pp. 117–131. abstractions,” in Proceedings of the 1st International Workshop
19. M.R. Genesereth, “The use of design descriptions in automated on Principles of Diagnosis, Stanford, CA, July 1990, pp. 102–
diagnosis,” Artificial Intelligence, vol. 24, pp. 411–436, 1984. 111.
20. A. Goel, “Integration of case-based reasoning and model-based 38. M. Quillian, “Semantic memory,” Semantic Information Pro-
reasoning for adaptive design problem solving,” Ph.D. Thesis, cessing, edited by M. Minsky, MIT Press: Cambridge, Mass.,
Department of Computer and Information Science, Ohio State pp. 227–353, 1972.
University, 1989. 39. S.A. Rajamoney and H.Y. Lee, “Prototype-based reasoning: An
21. A.R. Golding and P.S. Rosenbloom, “Improving rule-based sys- integrated approach to solving large novel problems,” in Pro-
tems through case-based reasoning,” in Proceedings of the Ninth ceedings of the Ninth National Conference on Artificial Intel-
National Conference on Artificial Intelligence, AAAI Press, ligence, AAAI Press, MIT Press: Anaheim, CA, July 1991,
MIT Press. July 1991, pp. 22–27. pp. 34–39.
22. F. Gomez and B. Chandrasekaran, “Knowledge organization and 40. M. Redmond, “Combining case-based reasoning, explanation-
distribution for medical diagnosis,” IEEE Transactions on Sys- based learning and learning from instruction,” in Proceedings of
tems, Mans and Cybernetics, vol. 11, pp. 34–42, 1981. the Sixth International Workshop on Machine Learning, Morgan
23. T. Govindaraj, “Quantitative simulations of complex dynamic Kaufmann: Ithaca, New York, 1989.
systems: An application to a marine power plant under supervi- 41. M. Redmond, “Distributed cases for case-based reasoning:
sory control,” in Proceedings of the Nineteenth Annual Confer- Facilitating use of multiple cases,” in Proceedings of the Na-
ence on Manual Control, Cambridge, MA, May 1983, pp. 377– tional Conference on Artificial Intelligence (AAAI-90), Morgan
390. Kaufmann: Boston, MA, 1990.
24. T. Govindaraj and Y.L. Su, “A model of fault diagnosis per- 42. R. Reiter, “A theory of diagnosis from first principle,” in Artificial
formance of expert marine engineers,” International Journal on Intelligence, vol. 32, pp. 57–95, 1987.
Man Machine Studies, vol. 29, pp. 1–20, 1988. 43. C. Riesbeck and R. Schank (eds.), Inside Case-Based Reasoning,
25. K.J. Hammond, “CHEF,” in Inside Case-Based Reasoning, Lawrence Erlbaum Associates, 1989.
edited by C. Riesbeck and R. Schank, Lawrence Erlbaum Asso- 44. R.C. Schank, “The structure of episodes in memory,” in Rep-
ciates, 1989. resentation and Understanding, edited by D.G. Bobrow and A.
26. F. Hayes-Roth, D.A. Waterman, and D.B. Lenat, Building Expert Collins, Academic Press: New York, pp. 237-272, 1975.
Systems, Addison-Wesley, 1983. 45. R.C. Schank, Dynamic Memory: A Theory of Reminding and
27. T.R. Hinrichs, “Strategies for adaptation and recovery in a design Learning in Computers and People, Cambridge University
problem-solver,” in Proceedings of the DARPA Workshop on Press, 1982.
P1: SJJ/AYV P2: SJJP1: SJJ/AYV P2: SJJ
Applied Intelligence 04-Feret January 6, 1997 14:37

78 Féret and Glasgow

46. V. Sembugamoorthy and B. Chandrasekaran, “Functional rep- interests include case-based reasoning, machine learning as well as
resentation of devices and compilation of diagnostic problem- real-time operating systems and multimedia systems.
solving systems,” Tech. Rep., Ohio State University, Colombus, He was a research assistant at Queen’s University in 1988–89 and
Ohio, 1985. worked at Spectrum Engineering Corporation Ltd., Peterborough,
47. S. Slade, “Case-based reasoning: A research paradigm,” in AI Ontario, in 1989–92 on the design and implementation of a real-time
Magazine, vol. 12, no. 1, pp. 42–55, 1991. monitoring and diagnostic system for robotic devices. He worked at
48. E.P. Sycara, “Resolving adversarial conflicts: An approach to Bell-Northern Research, Ottawa, Ontario in 1993–94 in the area of
integrating case-based reasoning and analytic methods,” Ph.D. real-time operating systems. He is now a member of Sony Corporate
Thesis School of Information and Computer Science, Georgia Research Laboratories in Tokyo, Japan, where he conducts research
Institute of Technology, 1987. and development projects in the area of on-line multimedia services
49. E. Tulving, Elements of Episodic Memory, Oxford University, for the Internet and for broadband networks. He also represents Sony
Oxford, UK, 1983. at the Digital Audio Visual Council (DAVIC).
50. W.C. Yoon and J.M. Hammer, “Deep-reasoning fault diagnosis:
An aid and a model,” in IEEE Transactions on Systems, Man,
and Cybernetics, vol. 18, no. 4, pp. 659–675, 1988.

Janice Glasgow is a Professor in the Department of Computing


and Information Science. She received her B.Sc. degree from the
Michel P. Féret received his B.Sc. degree in computer science from University of Alberta and M.Math and Ph.D. from the University
the University Pierre et Marie Curie, Paris 6, France in 1987, and his of Waterloo. Dr. Glasgow’s main area of interest and expertise is
M.Sc. and Ph.D. degrees in computer science from Queen’s Uni- Artificial Intelligence, and in particular computational imagery and
versity, Kingston, Ontario, Canada, in 1988 and 1993. His research its application to molecular scene analysis.