See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/221003080

A Survey on Ontology Metrics

Conference Paper · September 2010

DOI: 10.1007/978-3-642-16318-0_4 · Source: DBLP

CITATIONS READS

19 1,010

3 authors:

Juan Garcia Francisco José García-Peñalvo

Universidad Veracruzana Universidad de Salamanca
49 PUBLICATIONS   713 CITATIONS    1,010 PUBLICATIONS   7,228 CITATIONS   

SEE PROFILE SEE PROFILE

Roberto Therón
Universidad de Salamanca
205 PUBLICATIONS   1,153 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Visibility and impact of sciwntific grey literature in open access repositories View project

DEFINES View project

All content following this page was uploaded by Francisco José García-Peñalvo on 02 June 2014.

The user has requested enhancement of the downloaded file.

 A Survey on Ontology Metrics
J. García, F, J. García-Peñalvo and R. Therón

Computer Science Department. Science Education Research Institute (IUCE). GRIAL

Research Group. University of Salamanca
University of Salamanca, Spain
{ganajuan, fgarcia, theron}@usal.es

Abstract. Ontologies have been widely used in almost any field of application.
The use of ontologies should involve the possibility of evaluation of the quality
and correctness of them. Some tools and metrics have been proposed to reach
this goal. ONTOMETRIC, OntoQA and Protégé represent the most important
tools to evaluate ontologies. On the other hand, diverse cohesion, coupling and
ranking metrics have also been proposed, as well as methodologies such as
OntoClean. This paper analyses these tools and metrics, compare them and
finally reviews the current state-of-the-art concerning to ontology metrics.

Keywords: Ontology metrics, OWL, ontology cohesion, ontology coupling and

ontology ranking.

1 Introduction

In general, an ontology describes formally a domain of discourse. Typically, an

ontology consists of a finite list of terms and the relationships among these terms.
Ontologies play an important role to provide shared knowledge models to semantic-
driven applications targeted by Semantic Web. Ontology metrics represent an
important approach due to they can help to assess and qualify an ontology. From the
viewpoint of ontology developers, by assessing quality of ontology, they can
automatically recognize areas that might need more work and specify some parts of
the ontology that might cause problems. Furthermore metrics are useful in the process
of reuse because before using a previously defined ontology would be desirable to
evaluate it in order to determine the worthiness of using it. Metrics should always be
taken into account to evaluate ontologies both during engineering and application
processes. This paper starts with a brief introduction of ontologies and metrics, and
then the second section includes some metric definitions. The third section includes
an overview of the tools that implement some metrics. The fourth section makes a
comparative over the different metrics and finally the conclusions are discussed.
2 Metric Definitions

Diverse ontology metric proposals have been done in the past years; such as [6] that
describes some metrics to normalize ontologies and [7][12] that represent a way to
rank them. The paper [6] reviews the current state-of-the-art and basically proposes
normalization as a pre-process to apply structural metrics. This normalization process
consists of five steps: name anonymous classes, name anonymous individuals,
classify hierarchically and unify the names, propagate the individuals to the deepest
possible classes and finally normalize the object properties. This proposal is focused
on content metrics based on OntoMetric framework and basically they have been
proposed to improve ontology behavior or to fix some mistakes. Papers [7][12]
propose some metrics to rank ontologies. Basically this proposal consists of a Java
Servlet to process as inputs some keywords introduced by the user. Then the
framework searches using Swoogle1 engine and retrieves all the URI's representing
the ontologies related with these keywords. Then the framework searches on its
internal database if these ontologies have been previously analyzed and retrieves their
information. Finally the framework ranks retrieved ontologies.
Orme et al. [10] proposed a set of coupling metrics for ontology-based systems
represented in OWL, these metrics are: the number of external classes (NEC),
reference to external classes (REC), and referenced includes (RI). This proposal
defines a new type of coupling measurement for system development that defines
coupling metrics based on ontology data and its structure. The first proposed metric is
NEC, representing the number of distinct external classes defined outside the
ontology but used to define new classes and properties in the ontology. The external
classes can include standard classes defined as ontology language primitives and user-
defined classes from other ontologies. The second metric REC is the number of
references to external classes in the ontology. As we described above, NEC is a direct
measure of the number of classes in the ontology. REC is a direct measure of the
number of fanouts (in this case fanouts are different class hierarchies with external
roots) within the ontology resulting from external classes. RI is a direct measure of
the number of referenced includes in the ontology. Authors have also proposed some
cohesion metrics for ontologies [13]. They proposed a set of ontology cohesion
metrics to measure the modular relatedness of OWL ontologies. These metrics are
Number of Root Classes (NoR), Number of Leaf Classes (NoL) and Average Depth
of Inheritance Tree of all Leaf Nodes (ADIT-LN). Authors define NoR metric as the
total number of root classes explicitly defined in the ontology. A root class in an
ontology means the class has no semantic super class explicitly defined in the
ontology. NoL metric is defined as the number of leaf classes explicitly defined in the
ontology. A leaf class in an ontology means the class has no semantic subclass
explicitly defined in the ontology. Finally ADIT-LN is defined as the sum of depths
of all paths divided by the total number of paths. A depth is the total number of nodes
starting from the root node to the leaf node in a path. The total number of paths in an
ontology is all distinct paths from each root node to each leaf node if there exists an
inheritance path from the root node to the leaf node. And root node is the first level in
each path.

1 http://swoogle.umbc.edu
 Yinglong et al. [14] proposed another set of ontology cohesion metrics to measure
the modular relatedness of ontologies in the context of dynamic and changing Web.
These metrics have been defined taking into account the cohesion principle from
Object Oriented Approach adapted to ontologies. Authors concentrate on measuring
inconsistencies in ontologies and fully consider the ontological semantics rather than
structure. The metrics they propose are Number of Ontology Partitions (NOP),
Number of Minimally Inconsistent Subsets (NMIS) and the Average Value of Axiom
Inconsistencies (AVAI). This work also describes the algorithms to compute these
metrics and validate the metrics by using validation frameworks. These metrics are
focused on assessing the quality of ontologies. Authors define NOP metric as the
number of semantical partitions of a knowledge base. NMIS is defined as the number
of all minimally inconsistent subsets in a knowledge base. This metric is useful to
measure the scope of inconsistency impacts of a knowledge base. The third metric
AVAI is defined as: the ratio of the sum of inconsistency impact values of all axioms
and assertions to the cardinality of the knowledge base. Moreover, the article analyses
and validates the proposed metrics. Generally speaking, the advantages of these
metrics include the possibility of assessing the quality of a consistent ontology.
OntoClean methodology [4], [5] proposes the use of some defined metaproperties.
These metaproperties are rigidity, unity, identity and dependency. Authors have
borrowed these concepts from their ancient philosophical counterparts. The
methodology consists of assigning these metaproperties to the entities in order to
provide with a logical and semantic meaning. Applying these metaproperties results
on imposing several constraints on the taxonomic structure of an ontology and let to
develop a conceptual analysis of the concepts and their validity. Moreover this
methodology let to analyze and detect not logically consistent relationships.
YANG et al. [15] proposed metrics from a different point of view taking into
account the evolution of the ontologies. Authors suggest a metrics suite of
complexity, which mainly examine the quantity, ratio and correlativity of concepts
and relationships, to evaluate ontologies from the viewpoint of complexity and its
evolution. These metrics are divided into two groups: Primitive Metrics and
Complexity Metrics. The Primitive metrics include TNOC (Total Numbers of
Concepts or Classes), TNOR (Total Number of Relations), TNOP (Total Number of
Paths), where a path is defined as a trace that can be taken from a specific particular
concept to the most general concept in the ontology. The first Complexity Metric
defined is the average relations per concept that is calculated by dividing TNOR by
TNOC. The second metric is the average paths per concept, and is calculated by
dividing TNOP by TNOC.

3 Tools that implement metrics

There are some developed tools that implement diverse metrics. ONTOMETRIC,
OntoQA and Protégé represent the main available proposals. We consider that
OntoQA [8][9] represents the main proposal about metrics on ontologies. It proposes
some Schema Metrics to measure the richness of schema relationships, attributes and
schema inheritance. These metrics are focused on evaluating the ontology in general.
Other proposed categories are class richness, average population, cohesion,
importance of a class, fullness of a class, class inheritance and class relationship
richness, connectivity and readability. Class Relationship Richness is defined as the
number of relationships that are being used by instances that belong to the class. On
the other hand, the Connectivity of a class is defined as the number of instances of
other classes that are connected to instances of the selected class. All these metrics are
focused on the structure of the ontology.
Currently Protègè2 is the most widely used tool to create or modificate an ontology.
The metrics are classified into 6 categories. The first one is related with general
metrics such as counters for classes, object properties, datatype properties and
individuals. The second category is related with class axioms and includes counters
for subclass axioms, equivalent class axioms, disjoint classes axioms, GCI and hidden
GCI. The third category includes counters for object properties axioms. These
counters are total of sub object properties, equivalent, inverse, disjoint, functional,
inverse functional, transitive, symmetric, anti symmetric, reflexive and irreflexive
object properties. Furthermore object property domain and range counters are also
included. The fourth category is dedicated for datatype properties counters. This
category includes total values for sub datatype properties, equivalent, disjoint and
functional datatype properties, as well as counters for data properties domain and
range. The fifth category is focused on individuals. It defines counters for class
assertions, object and datatype property assertions, negative object and negative
datatype assertions and same or different individual axioms. Finally, the last category
involves annotation axioms and defines just two metrics, the entity annotation axioms
count and the axiom annotation axioms count. All these metrics represent simple
counters for the items in the ontology and do not provide any kind of semantic metric.
ONTOMETRIC [11] is a framework proposed to measure the suitability of
existing ontologies. This tool was defined to quantify the suitability of ontologies.
Authors propose a taxonomy of 160 characteristics, also called multilevel framework
of characteristics that provides the outline to be able to choose and to compare
existing ontologies. This framework is used as a representation template of the
information and starts by defining an analytic hierarchy process. This process
involves building a hierarchy tree with the root node being the objective of the
problem. The intermediate are the criteria and finally the lowest levels contain the
alternatives. Then as the second step, the methodology applies the analytic hierarchy
process to decide whether or not to reuse ontologies.
Ontology Metrics 3 is a web-based tool that validates and displays statistics about
an OWL ontology, including the expressivity of the language it is written in. This tool
calculates the same metrics than Protégé. These metrics include counters for classes,
properties, individuals, logical axioms, as well as specific counters described above in
the Protégé section.

2 Protégé Ontology Editor http://protege.stanford.edu/

3 http://owl.cs.manchester.ac.uk/metrics/
4 Comparing metrics

Comparing the diverse analyzed metrics, we consider that OntoQA offers the best set
of metrics for analyzing the structure and ranks of the ontologies according to these
metrics. Just Yinglong proposal and OntoClean are clearly focused on the semantics.
On the other hand Orme, Yinglong and OntoQA have proposed different metrics to
calculate the cohesion. It is important to highlight that they represent three completely
different ways to define cohesion. OntoQa defines cohesion as the number of separate
connected components of the graph representing the Knowledge Base. In contrast
Orme's proposal metrics measure the modular relatedness of OWL ontologies while
Yinglong focuses on measuring ontologies in the context of dynamic and changing
web. The metrics proposed by Orme et al. [10] represent the coupling among entities
from diverse ontologies. There is another coupling that has not been taken into
account: the coupling among entities in the same ontology that provides useful
information about the connection among the classes in the same ontology. Sometimes
systems use no more than one ontology then the coupling metrics proposed by Orme
become completely useless. The metrics proposed by Yang et al. [15] are intended to
reflect the complexity of an ontology during its lifecycle and evolution.

Table 1. Summary of the analyzed metrics and comparison of some interesting properties.

Metric Semantic / Structure Ranking Cohesion Coupling

Vrandecic Structure No No No
Alani Structure Yes No No
Orme Structure No Yes Yes
Yinglong Semantic No Yes No
OntoClean Semantic No No No
Ontometric Structure No No No
Protégé Structure No No No
OntoQA Structure Yes Yes No
Ontology Structure No No No
Metrics
Yang Structure No No No

5 Conclusions and future work

Most of the current metrics and tools are focused on the evaluation of structure of the
ontologies such as Protégé, Ontology Metrics tool or OntoQA. Other metrics are
focused on cohesion such as NoR, NoL, ADIT-LN, NOP, NMIS and AVAI. There
are a few of them focused on coupling such as NEC, REC and RI. OntoClean
represents an interesting approach based on ontologies semantics. Moreover some
metrics are defined to rank ontologies, normalize them or to qualify them as
ONTOMETRIC. A completely different approach is proposed by Yang, which is
based on measuring the complexity of an ontology taking into account the evolution.
We have analyzed diverse metrics for ontologies. Furthermore we have also provided
with a classification based on different aspects such as semantics or structure. Finally
we can conclude that diverse metrics have been proposed to cover diverse aspects to
evaluate, being most of them focused on the structure instead of semantics. Our future
work includes the definition of some metrics focused on coupling among entities
within the same ontology, based on the semantics.

Acknowledgements

This work was supported by the Spanish Industry Ministry (project TSI-020302-
2009-35) and by the Castile and Lion Regional Government through GR47 excellence
project.

References

1. Konstantinos Makris, Nikos Bikakis, Nektarios Gioldasis, Chrisa Tsinaraki and Stavros
Christodoulakis, "Towards a Mediator Based on OWL and SPARQL", Visioning and
Engineering the Knowledge Society. A Web Science Perspective, pp. 326-335, Springer
Berlin 2009
2. Alberto Salguero, Cecilia Delgado and Francisco Araque, "STOWL: An OWL Extension for
Facilitating the Definition of Taxonomies in Spatio-temporal Ontologies", Visioning and
Engineering the Knowledge Society. A Web Science Perspective, pp. 326-335, Springer
Berlin 2009
3. Grigoris Antoniou and Frank van Harmelen, “A Semantic Web Primer”, second edition, The
MIT Press, Cambridge Massachusetts, 2008
4. Nicola Guarino and Chris Welty, "An Overview of OntoClean", The Handbook on
Ontologies, Pp. 151-172, Berlin:Springer-Verlag, 2004
5. Nicola Guarino and Chris Welty, "Evaluating Ontological Decisions with OntoClean",
Communications of the ACM, pp 61-65, ACM Press, 2002
6. Denny Vrandecic and York Sure, "How to Design Better Ontology Metrics", The Semantic
Web: Research and Applications, pp. 311-325, Springer-Berlag 2007
7. Harith Alani, Christopher Brewster and Nigel Shadbolt, "Ranking Ontologies with
AKTiveRank", Proceedings of the International Semantic Web Conference, ISWC, 2006
5th International Semantic Web Conference (ISWC), November 2006, Georgia, USA
8. Samir Tartir, Budak Arpinar, Michael Moore, Amit Sheth and Boanerges Aleman-meza,
"OntoQA: Metric-based ontology quality analysis", CiteSeerX - Scientific Literature Digital
Library and Search Engine, 2005
9. Samir Tartir and Budak Arpinar, "Ontology Evaluation and Ranking using OntoQA",
Proceedings of the International Conference on Semantic Computing, 2007
10. Anthony Orme, Haining Yao, and Letha Etzkorn, "Coupling Metrics for Ontology-Based
Systems", IEEE Software, pp 102-108, 2006
11. Adolfo Lozano-Tello and Asunción Gómez-Pérez, "ONTOMETRIC: A Method to Choose
the Appropiate Ontology", Journal of Database Management, 2004
12. Harith Alani and Christopher Brewster, "Metrics for Ranking Ontologies", 4th Int. EON
Workshop, 15th Int. World Wide Web Conference, 2006
 13. Haining Yao, Anthony Orme and Letha Etzkorn, “Cohesion Metrics for Ontology Design
and Application”, Journal of Computer Science 1(1): 107-113, 2005, Science Publications
14. Yinglong Ma, Beihong Jin, Yulin Feng, "Semantic oriented ontology cohesion metrics for
ontology-based systems", The Journal of Systems and Software, Elsevier, 2009
15. Zhe YANG, Dalu Zhang and Chuan YE, "Evaluation Metrics for Ontology Complexity and
Evolution Analysis", IEEE International Conference on e-Business Engineering
(ICEBE'06), 2006

View publication stats