FORMATIVE EVALUATION OF ONTOLOGIES FOR INFORMATION AGENTS Muthukkaruppan Annamalai Faculty of Computer Science and Software Engineering, MARA

University of Technology, 40450 Shah Alam, Selangor, Malaysia (mk@tmsk.uitm.edu.my)

ABSTRACT Information agents require knowledge about a particular domain to gather useful information for humans. This knowledge should ideally be stored externally in ontologies. An application developer either builds or simply adopts and extends existing domain ontologies to describe the applications resources in meaningful ways to information agents. In order to realise this potential, the domain knowledge should be explicitly represented within `good ontologies. The question is how do we know if an ontology being evolved is good, i.e., how do we know if an ontology is appropriate for particular need? If not, what is lacking in it and how it can be enhanced to make it suitable for use? The answer emerging in the knowledge engineering literature is ontology evaluation. The evaluation of an ontology is crucial for its successful use and reuse, yet it is acknowledgingly a difficult issue to tackle. Up to now few works have been performed to this end. This paper addresses the issue of formative evaluation (validation and verification) of the ontologies being developed for information agents. In particular, it critically reviews the existing criteria based evaluation practices to advance a more feasible set of criteria, making additional distinctions about the aspects of the ontology that came under the purview of each of the evaluation criterion. 1. INTRODUCTION Information agents can search and gather specific pieces of information from the distributed knowledge sources in a networked environment. The ability of an information agent to perform this task effectively rests with the knowledge it holds about the domain. This adaptable knowledge must be described in a way that the information agents can use during search and retrieval, and hence warrants an ontology. An ontology is a representation vocabulary, specialised in some domain or subject matter. The ontology defines a set of concepts for representing specific facts in an instance of a domain (Chandrasekaran and Josephson 1999). The emphasis is on the conceptualisation of the conceptual terms in the vocabulary intended to be captured. Consequently, the notion that ontology is an explicit specification of conceptualisation in a shared domain as proposed by Gruber (1993) is the widely cited definition in artificial intelligence (AI) and information systems (IS). Ontology serves as a primary source for structuring and annotating knowledge content, and subsequently provides machinereadable semantic knowledge about the information in the knowledge sources to the agent. A shared ontology additionally facilitates a knowledge community to exchange information in a domain. A good domain ontology represents the modelled world closely through consistently and coherently specified definitions of concepts and relationships that hold among them. It directs an information agent to infer precise information from the knowledge contents that are conceptually linked to this ontology. In contrast, an ontology containing erroneous and incomplete definitions does not provide a reliable semantic basis for applications that depend on it. Consequently, a formative evaluation of an ontology can help to identify its inherent deficiencies that can be addressed while the ontology is being developed to produce an effective conceptualisation of the knowledge in the domain of discourse. Unfortunately, the research and development on shared domain ontologies for information agents is still largely confined to the academic environment. Since researchers are more concerned about the technology that utilises an ontology, passable ontologies are seen as sufficiently meeting their current research needs. As such, evaluating the goodness of domain ontologies has not been a primary interest of researchers. This is evident from the very small number of papers that have been written about ontology evaluation so far. However, we believe this lackadaisical attitude is set to change with the advent of agents to facilitate knowledge enhanced search and information retrieval. This will spur the production of good domain ontologies for a wide range of application domain. Formative evaluation plays a key function in the development of a good ontology. The evaluation corroborates the reliability of the ontology being built, i.e., whether the conceptual definitions in the ontology are well-founded. If not, the evaluation effort prompts us to consider what is needed to produce an effective conceptualisation that could adequately represent the knowledge in the domain of discourse. A domain ontology is often designed with the potential of reuse in various applications in a domain area. Therefore, a seemingly time saving strategy to create a new ontology for describing a specific situation in a domain is by simply adopting an existing one. However,

in reality the domain specific knowledge needed by an agent varies from application to application. Some applications require more or less detailed knowledge about a domain than others. As such, the evaluation of the selected ontology is a precursory step to decide whether that ontology is suitable for adoption. The formative evaluation will expose the ontology's strengths and weaknesses, and suggests the modifications and extensions that are necessary to make the ontology amenable to use. In principle, an ontology should be evaluated on the syntax, structure and semantics of its conceptual definitions. The syntactic correctness of the definitions could be checked using syntax analysers incorporated within editors, parsers and validators. There has been considerable work developing such tools. Discussing them is beyond the scope of this paper, but information about some of these tools can be found at ontology language websites such as http://www.daml.org/tools/ tools.html/ and http://www-ksl-svc.stanford.edu/. During the development of an ontology, we are mainly concerned with the evaluation of the structure and semantics of the conceptualised entities. In this paper, we present a set of criteria to assess the structural and semantic adequacy of a domain ontology in the context of its applications in a domain area. In Section 2, we give a brief description about ontologies for information agents. In Section 3, we critically review the existing criteria based evaluation practices to propose a more feasible set of ontology evaluation criteria. The evaluation of ontologies with respect to the prescribed set of evaluation criteria is reviewed in Section 4. Finally, the concluding section we summarise the contribution of this paper and point to the direction of future work. 2. ONTOLOGIES FOR INFORMATION AGENTS An ontology for information agents is a library of related concepts, explicitly defined and formally organised into subclasses, as a way of structuring and defining the meaning of the represented conceptual terms. In general, the ontology specifies classes of entities (concepts) in the domain, relationships between these concepts and properties attributed to them. Rigorously structured ontologies also describe functions and axioms (rules) associated with these concepts to further constrain their interpretation. In the simplest case, however, an ontology is structured as a hierarchy of concepts related by subsumption (subtype-supertype) relationships, reflecting a taxonomy of conceptualisation. 3. RELATED WORK AND DISCUSSION The importance of ontology evaluation is recognised in existing literature on ontology development. At the same time, it is commonly acknowledged that evaluating the quality of an ontology is a difficult issue to tackle.

Gruninger and Fox (1995) and, Uschold and King (1995) highlight the evaluation of the reliability of a developed ontology. The former uses a set of competency questions to evaluate the suitability of an ontology, while in the latter, the ontology is evaluated against its requirement specification. The importance of ontology evaluation is also stressed by Fernandez et. al. (1997), except that the evaluation is to be carried out as part of the ontology development process; this paper shares a similar view. A more serious effort to evaluate ontologies was initiated by Gomez-Perez (1996). Drawing inspiration from evaluation of knowledge systems, she emphasises both validation and verification of the ontology content are two important aspects of ontology evaluation. In retrospect, the origin of these terms can be traced to evaluation of software (Boehm 1981), the precursor for knowledge systems. Validation and verification are associated with checking the appropriateness and the correctness of the developed software, respectively. Adapted to the evaluation of ontologies, validation checks the suitability or appropriateness of the ontology being designed, and verification checks the correctness and the adequacy of the ontology. The ontology is evaluated against its frame of reference to ensure that it appropriately and adequately satisfies its purpose of design. Without such frames of references, it is difficult to check if the ontology is in compliance with its purpose of design, i.e., conforms to the understanding of what it is to be used for, and competently satisfies its needs and requirements. The common frames of references are requirement specification, competency questions and, knowledge and information resources in the subject domain. Subsequently, the researchers in ontology development have recommended numerous evaluation criteria, on which the technical evaluation of the ontologies can be based in order to guide the development of ontologies having requisite quality of design. In essence, the evaluation is a subjective means to check the compliance of the ontology with respect to certain desired properties in ontologies. From the perspective of modelling in AI, Gruber (1995) advocated the use of criteria such as clarity, coherence, extensibility, minimal encoding bias, and minimal ontological commitment, which serve as precedent for the design of an ontology. These notions reverberate in contemporary ontology development methodologies. For example, Uschold (1996) prescribed that an ontology is evaluated using criteria as clarity, consistency and reusability, which in retrospect, a set of design criteria. In the same vein, Fox and Gruninger (1998) suggested the use of criteria as perspicuity, precision, generality, granularity, minimality (conciseness), expressiveness and competence. GomezPerez (1999, 2001) proposed consistency, completeness, conciseness and expansibility as worthy evaluation criteria.

The modelling of knowledge in AI is strongly similar to the modelling of the entities and relationships in the conceptual schema. From the perspective of IS, the entity-relationship model alludes to design qualities as correctness, completeness, understandability, simplicity, integration and implementability (Moody and Shanks 1994). The aforementioned design qualities cover much of the same kind of ground as the evaluation criteria mentioned in the previous paragraph. It is obvious that the above researchers tend to distinguish different set of criteria based on different concerns in the design and evaluation of an ontology. In spite of that, the quality demanded of some criteria overlap with many others. We cite the following as examples: Coherence and consistency correspond with correctness; Clarity extends along with understandability, perspicuity and expressiveness; Granularity and precision accede to competency; Minimal ontological commitment, generality and reusability are conformable criteria. However, accepting all of the above as valid design criteria will give rise to conflict. For example, the admittance of the level of expressiveness offered by rigorous formalism for proper interpretation of defined concepts, clashes with a simultaneous call for simplicity. Likewise, there is a friction between clarity and conciseness. The discord between reusability of the modelled knowledge and its usability is another matter to take issue. We deem it is essential to bring together a consolidated set of evaluation criteria that are compatible with each other. Of the utmost necessity is the conceptual definitions must be semantically consistent and coherent with the modelled area of knowledge, and the definitions must be consistent with each other, i.e., none of the conceptual definitions in the ontology contradicts with the other. Since, global consistency is not easy to enforce, we should at least strive to achieve local consistency, i.e., an ontology that is consistent with the view of a particular user-community. An effective ontology must be adequately complete to satisfy its needs and requirements. An ontology void of unnecessary details is free from ambiguity and redundancy, and additionally facilitates its checking and maintenance. Because ontology development is an evolutionary process, the design must allow for incremental modifiability, i.e., it must be able to be monotonically extended, conserving the existing conceptualisation. Therefore, we regard consistency and coherence, completeness, conciseness and, extensibility and expansibility as important evaluation criteria of ontologies. A useful ontology should be sufficiently expressive, whose concepts are defined at the appropriate level of details and is adequately competent in meeting its purpose of design. Representing the conceptualised model with minimal encoding bias is yet another practical design criteria. Hence, we set down expressibility, minimal encoding bias and competency as additional evaluation

criteria, which brings about the following discussion on the impending design trade-offs. On the trade-off between simplicity and expressibility, we concur with the general view that ontologies should be kept simple so that they are easy to implement that encourages its reuse (Staab 2002). Nevertheless, we will still argue that an ontology must be sufficiently expressive to appropriately constrain the possible interpretation of the conceptual terms that meets its original purpose of design. We also concur with Gruber that an ontology should as far as possible be represented independent of symbollevel encoding. Representing an ontology at an intermediate level apart from its impending implementation allows for the specification of conceptualisation that is not severely restricted by particular language limitation. In relation to competency, there is a trade-off between reusability and usability of an ontology. The modelling for usability is directed by a purposive mechanism, so that a defined concept can be applied more directly to specify related knowledge content. On the other hand, the modelling for reusability emphasises generically represented vocabulary that is applicable across many domains in a variety of situations. Ontologies developed following the latter fashion of design assert minimal ontological commitments, thus calling for another design decision to make the necessary extension and rework before they can be usefully applied. Further more, the generic terms in the extended ontology tend to cloud the conceptualisation with extraneous knowledge that may be of little or no interest to the knowledge community. Therefore, we argue that working towards adequate ontological commitment to support the desired competency should be viewed as an integral aspect of ontology design, such that the ontology is at disposal for use. Hence, we propose consistency and coherence, completeness, conciseness, extensibility and expansibility, competency, expressiveness and minimal encoding bias as a credible set of ontology evaluation criteria for information agents. 4. ONTOLOGY EVALUATION CRITERIA In this section, we discuss the evaluation of ontologies with respect to the prescribed set of ontology evaluation criteria; and making additional distinctions about the aspects of the ontology that came under the purview of each of the evaluation criterion. 4.1. Consistency and Coherence The most important criteria in ontology evaluation are to verify the consistency and coherence of the conceptual definitions in the ontology. The verification checks on the correctness and relevance of the concepts and the properties associated with them. It involves two levels of

analyses. Firstly, we analyse the structural integrity of individual conceptual definitions to check if each concrete concept in the ontology corresponds to a specific entity in the modelled area of domain knowledge, and the definition is logically consistent with the intended conceptualisation. We also check to ensure that each individual assertion (instantiated concept) in the ontology is true. Secondly, we analyse the definitions in the ontology as a whole, which includes the abstract concepts. Collectively, the definitions in the ontology must also be consistent and coherent with each other, such that the definitions in the ontology do not contradict with one another. 4.1.1. Individual Conceptual Integrity An ontology is comprised of a restricted set of concepts related to an area of knowledge. For each concept, a set of properties, i.e., attributes and relations linking it to other concepts are defined to facilitate its interpretation. Information is used to crystallise a concrete concept's structure with the appropriate semantic content such that an instantiated concept can carry specific information about the entity it represents. The conceptual integrity verifies the individual conceptual definitions with respect to the information used to articulate a concept's structure. We also check to ensure that there is no contradiction in the interpretation of a concrete concept with respect to the entity it represents. 4.1.2 Collective Consistency Collective consistency analyses the coherence of the definitions in the ontology by verifying the relationships that bind the concepts. We verify the specialisation, generalisation and equivalence relationships depicted in subsumption hierarchical classification, the commonly used scheme for structuring concepts in ontologies. We also review the association or cross relationships that exist between the concepts in the ontology, as well as the reciprocal relationships that complement the cross relationships such as inverse and transitive relationships. The idea is to check whether the logical relationships defined in the ontology intuitively reflect the dependencies between their corresponding entities (both concrete and abstract) in the modelled area of knowledge. Like the concept structures, the relations can also be structured in the ontology. A relation can be defined as a specialised or generalised form of another relation; or it can be specified as equivalent-to or inverse-of another relation. In such case, the relational dependencies ought to be analysed separately. 4.2. Completeness If consistency and coherence criteria are used to verify the correctness and relevancy of the concepts and the

properties defined in the ontology, the completeness criterion checks whether the ontology has covered all the relevant concepts and properties. Gomez-Perez (1999) goes a step further by stating that an ontology is deemed complete if each definition in the ontology is complete with respect to the real world. This additional requirement is voiced in a philosophical tone and demands that each definition in the ontology is analysed with respect to its coverage of the knowledge in the real world -- a requirement that is hard to accomplish. Ontologies are by nature incomplete; and we uphold the view that a domain ontology is not a representational mechanism to answer arbitrary questions about the domain. Based on the above argument, our completeness evaluation will go as far as to assess the functional adequacy of the ontology in the context of its use and the purpose of its design. The functional needs are stipulated in the requirement specification and described by a pregenerated set of competency questions. So, when we say an ontology is complete, we actually mean the ontology is functionally complete with respect to the area of knowledge implied by its reference framework, rather than ontologically complete with respect to the real world. We will see later that a functionally complete ontology is the basis for evaluating the competence of an ontology. A functionally complete ontology meets all the requirement specification of the modelling needs and/ or is able to characterise the answers of all the competency questions using the terms defined in the ontology. The competency questions play an eminent role in the completeness analysis of the modelled purposive domain knowledge. The ontology needs to support not only the statement of the answer, but also the derivation of the answer for each of the competency questions. The incompleteness of an ontology becomes evident when either the requirement specifications are not satisfied or the ontology cannot express the answer(s) to a competency question posed to it. To address the deficiency in the ontology, the missing concepts and the properties have to be accounted for. 4.3. Conciseness In contrast to the verification of completeness, the verification of conciseness checks to see whether there are extraneous definitions in the ontology. This analysis seeks to identify and remove redundant definitions that are present in the ontology. In general, unnecessary and unwanted definitions do not add value to the ontology. Obvious redundancies are redefinition of existing concepts in an ontology. Particularly, redefinition of equivalent concepts using separate set of properties must not be allowed because they are susceptible to differing interpretation and use. The redefinitions that introduce polysemous concepts can seriously impair the reliability of the ontology. We believe such redundancy could be avoided by questioning the justification behind the inclusion of each definition in the ontology. A more

demanding task is to check for inferred redundancies in the ontology, i.e., whether a result can be deduced in more than one way. The point is to remove concepts and properties whose presence in the ontology cannot be justified. 4.4. Extensibility and Expansibility While an ontology carving a specific area of knowledge can be shaped as a separate entity, it must be easy to extend the ontology and expand its conceptual definitions. The idea is to allow for incremental modification, specialisation and adaptation of the ontology without having to revise the existing definitions. Extensibility checks to ensure that the concepts are defined and organised in the manner that facilitate future extension of the ontology. Expansibility checks if the ontology allows expansionary modification such as addition of new conceptual definitions and expansion of existing definitions without degenerating its present state of being. Clearly, the formalism in which an ontology is represented is a tangible factor determining its extensibility and expansibility. For example, the hierarchically organised, objectified concept definitions constructed using Frames provides a framework for ordered representation that are easy to read and change, and are also able to scale more easily as compared to say, logical rules (Grosso et. al. 1999). 4.5. Competency The competency criterion is specifically used to assess whether the concepts in the ontology are defined with the requisite level of detailness. The check on competency helps to ensure that an ontology is capable of supporting the purpose of its design. In general, the concepts in such ontologies ought to be represented at a finer level of granularity. The level of details that must be captured by a conceptual definition is however dependent upon the needs and requirements that arise in order to competently utilise the modelled knowledge. As a general rule, a competent ontology must also be functionally complete. The detail representation offers direct assistance for immediate adoption of the ontology. So, in addition to the verification of completeness with respect to the ontology's frame of reference, we also need to verify the competency of an ontology by checking on its ability to express relevant content of the knowledge artefacts. 4.6. Expressiveness and Minimal Encoding Bias As point out in Section 2, an ontology for information agents must be specified with a certain degree of formality. The ontology must be expressed using a formalism that is easy to read and understand so as to facilitate its evaluation, maintenance, implementation and

uptake. In other words, the ontologies must be accessible to both human and formal tools. Although expressiveness and minimal encoding bias are design decisions that is often factored into the ontology representation formalism, an intermediate, semiformal representation of the conceptual definitions that is as far as possible independent of particular symbol-level encoding offers greater vantage in the design and development of ontologies for information agents. 5. CONCLUSION The formative evaluation of an ontology for information agent encompasses validation and verification of the ontology content to corroborate the reliability of the ontology. The validation of the ontology is necessary to ensure that a `right' ontology is being built. For this, we propose to validate the ontology against its reference frames. Common frames of references are requirement specification, competency questions and, shared knowledge and information resources in the domain of discourse on which the development of the ontology was based. The verification of the ontology content is necessary to ensure that the ontology is built `right'. For this, we propose to check on the compliance of the ontology with respect to a feasible set of ontology evaluation criteria, namely consistency and coherence, completeness, conciseness, extensibility and expansibility, competency, expressiveness and minimal encoding bias. At present, ontology evaluation is performed manually. The question we ask is can suitable diagnostic tools be developed to facilitate the formative evaluation of ontologies. In addition to checking on the logical correctness of the ontological definitions, it is interesting to study to what extent completeness evaluation can be automated. For instance how can we gather, formalise and pose competency questions to ontologies using queryanswering tools in an organised manner? Conversely, could we make it possible for ontology development tools to systematically generate the competency questions that it is capable of answering, which can be easily verified by a knowledge engineer? Such supporting tools can largely alleviate the burden of competency evaluation. ACKNOWLEDGEMENT The author expresses special thanks and gratitude to Prof. Leon Sterling of The University of Melbourne for inspiring the research on ontology and for his comments on earlier writings on ontology design and evaluation.

REFERENCES
Boehm, B., 1981. Software Engineering Economics. PrenticeHall. NJ. Chandrasekaran, B and Josephson, J. R., 1999. What Are Ontologies, and Why Do We Need Them? In IEEE Intelligent Systems, January/February 1999. Fernandez, M., Gomez-Perez, A and Juristo, N., 1997. Methontology: From Ontological Art Towards Ontological Engineering. In Proceedings of AAAI97 Spring Symposium on Ontological Engineering, Stanford University, CA. Fox, M and Gruninger, M., 1998. Enterprise Modelling. In AI Magazine, 19(3). Gomez-Perez, A., 1996. A Framework to Verify Knowledge Sharing Technology. In Expert Systems with Application, 11(4). Gomez-Perez, A., 1999. Evaluation of Taxonomic Knowledge on Ontologies and Knowledge-Based Systems. In Proceedings of KAW99 Workshop on Knowledge Acquisition for Knowledge-Based Systems, Alberta. Gomez-Perez, A., 2001. Evaluation of Ontologies. In International Journal of Intelligent Systems, 16(3). Grosso, W., Eriksson, H., Fergerson, R., Gennari, J., Tu, S and Musen, M. Knowledge Modeling at the Millennium (The Design and Evolution of Protege-2000). In Proceedings of

KAW99 Workshop on Knowledge Acquisition, Modelling and Management, Alberta. Gruber, T., 1993. A Translation Approach to Portable Ontologies. In Knowledge Acquisition, 5(2). Gruber, T., 1995. Toward Principles for the Design of Ontologies Used for Knowledge Sharing. In International Journal of Human Computer Studies, 43(5). Gruninger, M and Fox, M., 1995. Methodology for the Design and Evaluation of Ontologies. In Proceedings of IJCAI95 Workshop on Basic Ontological Issues in Knowledge Sharing, Montreal. Moody, D and Shanks, S., 1994. What Makes a Good Data Model? Evaluating the Quality of Entity Relationship Models. In Proceedings of ER94 Conference on EntityRelationship Approach. Springer. Staab, S., 2002. Trends and Controversies: Ontologies KISSES in Standardization. In IEEE Intelligent Systems, March/April 2002. Uschold, M., 1996. Building Ontologies: Towards a Unified Methodology. In Proceedings of Expert Systems96 Conference of the British Computer Society Specialist Group on Expert Systems, Cambridge. Uschold, M and King, M., 1995. Towards a Methodology for Building Ontologies. In Proceedings of IJCAI95 Workshop on Basic Ontological Issues in Knowledge Sharing, Montreal.