Welcome to Scribd, the world's digital library. Read, publish, and share books and documents. See more
Download
Standard view
Full view
of .
Look up keyword
Like this
1Activity
0 of .
Results for:
No results containing your search query
P. 1
A Resemblance Study of Test Collections for World-altering Semantic Web Services

A Resemblance Study of Test Collections for World-altering Semantic Web Services

Ratings: (0)|Views: 5 |Likes:
Published by hsaboohi
Proceedings of the International MultiConference of Engineers and Computer Scientists 2011 Vol I, IMECS 2011, March 16 - 18, 2011, Hong Kong

A Resemblance Study of Test Collections for World-altering Semantic Web Services
Hadi Saboohi, Sameem Abdul Kareem
Abstract—Approaches to Web services discovery, matchmaking, composition, and execution monitoring need a collection of descriptions of test services. Semantic Web services help attain better accuracy in these approaches by clarifying service descriptions. In the semantic Web service research area, there are some test collections. These test collections have some information-providing services. However, there is not a proper test collection that contains a reasonable number of both information-providing and world-altering services. This paper intends to investigate current available test collections, as well as other sources of service description. Their features (such as their description and expression languages) and a number of world-altering services, preconditions, and effects are compared, and their usage in current contests and challenges are discussed. Finally, a road map for standard test collections for world-altering semantic Web services is provided.
Proceedings of the International MultiConference of Engineers and Computer Scientists 2011 Vol I, IMECS 2011, March 16 - 18, 2011, Hong Kong

A Resemblance Study of Test Collections for World-altering Semantic Web Services
Hadi Saboohi, Sameem Abdul Kareem
Abstract—Approaches to Web services discovery, matchmaking, composition, and execution monitoring need a collection of descriptions of test services. Semantic Web services help attain better accuracy in these approaches by clarifying service descriptions. In the semantic Web service research area, there are some test collections. These test collections have some information-providing services. However, there is not a proper test collection that contains a reasonable number of both information-providing and world-altering services. This paper intends to investigate current available test collections, as well as other sources of service description. Their features (such as their description and expression languages) and a number of world-altering services, preconditions, and effects are compared, and their usage in current contests and challenges are discussed. Finally, a road map for standard test collections for world-altering semantic Web services is provided.

More info:

Published by: hsaboohi on Sep 19, 2011
Copyright:Attribution Non-commercial

Availability:

Read on Scribd mobile: iPhone, iPad and Android.
download as PDF or read online from Scribd
See more
See less

09/05/2014

pdf

 
A Resemblance Study of Test Collections forWorld-altering Semantic Web Services
Hadi Saboohi, Sameem Abdul Kareem
 Abstract
—Approaches to Web services discovery, matchmak-ing, composition, and execution monitoring need a collectionof descriptions of test services. Semantic Web services helpattain better accuracy in these approaches by clarifying servicedescriptions. In the semantic Web service research area, thereare some test collections. These test collections have someinformation-providing services. However, there is not a propertest collection that contains a reasonable number of bothinformation-providing and world-altering services. This paperintends to investigate current available test collections, as well asother sources of service description. Their features (such as theirdescription and expression languages) and a number of world-altering services, preconditions, and effects are compared, andtheir usage in current contests and challenges are discussed.Finally, a road map for standard test collections for world-altering semantic Web services is provided.
 Index Terms
—Semantic Web Service, Test Collection, World-altering Services, Precondition, Effect
I. I
NTRODUCTION
L
IKE other scientific fields, specifically computer sci-ence, any evaluation needs test data. Test data are datathat have been explicitly generated or collected to be usedin evaluation exercises. This is typical for any computerprogram. Web services as remote applications in a service-oriented architecture (SOA) are not exempt from this issue.The test data for Web services are along two axes. First,the set of inputs to be checked by Web services (as remotemacro functions) if they are capable of generating the desiredoutputs. Second, is the description of services to be used inthe discovery and the matchmaking of intended service withthe composition of atomic services to fulfill a user’s goal.These descriptions can be used in execution monitoring andfailure recovery of atomic and composite services as well.This paper discusses the second type of test data for Webservices.“Semantics” as an add-on for Web service description canbe used to disambiguate the various definitions of functionaldescriptions of services. These functional descriptions of semantic Web services include input, output, precondition,and effect.Inputs and outputs describe information transformationdone by the service. Preconditions and effects describe thestate change of the world. Precondition is the state of theknowledge-world or real-world before the execution of thedescribed service. Effect is the description of the worldchange made by the service invocation.
H. Saboohi is a PhD student in the Department of Artificial Intelligence,Faculty of Computer Science and Information Technology, University of Malaya (UM), 50603 Kuala Lumpur, Malaysia.e-mail:saboohi@siswa.um.edu.myDr. S. Abdul Kareem is an Associate Professor in the Departmentof Artificial Intelligence, Faculty of Computer Science and InformationTechnology, University of Malaya (UM), 50603 Kuala Lumpur, Malaysia.e-mail:sameem@um.edu.my
Web services are categorized as
world-altering
servicesand
information-providing
services [1]. This categorization is determined by the presence or lack of a kind of effect bythe execution of the service. Any test data for semantic Webservices surely needs to contain both information-providingservices and world-altering services.Gathering a reasonable number of services generates acollection of test data, which is called a “
Test Collection
.”As it is common, these test collections may be producedeither manually, focusing on a relevant domain, or in anautomatic manner, in which a large-volume of randomizeddata is generated programmatically. In the semantic Webservice research field, both approaches have been used.For a practical Web service testing approach to be applica-ble in the real-world, it needs realistic test data. This testingapproach is for
mediation
techniques, including discovery,matchmaking, composition, and execution monitoring of Web services. Therefore, there is a critical need for actualstandard test collections of semantic Web services. Thesestandard test collections, like in any software developmentenvironment, are intended to be utilized to test mediationapproaches to show that it has the claimed ability.Another critical aspect in a test collection of semanticWeb services is the ontology of the concepts used in thedescription of semantic Web services. Functional specifica-tion of semantic Web services, particularly input and outputof a service, are described using taxonomies in an ontology.This is in contrast with the description of input and outputof traditional Web services. Traditional Web services arecommonly described using the Web Service DescriptionLanguage (WSDL)
1
. The input and output of traditional Webservices are described by the XML Schema
2
, which doesnot convey any meaning. Therefore, bundling a requiredontology (or some ontologies) is an important requirementfor a real, applicable test collection.In recent years, there have been a few efforts in making testcollections for semantic Web services. These collections areused in testing mediation approaches of semantic services.Some of these test collections are specifically created forvarious contests and challenges in the field of semanticservices (cf. to SectionV). A test collection of semantic Web services definitelyrequires a combination number of information-providing ser-vices and world-altering services. Currently, test collectionslack the fair number of each of these categories. This paperintends to analyze existing test collections in terms of theirdiverse features, and to provide a road map for standard testcollections of world-altering semantic Web services.
1
2
Proceedings of the International MultiConference of Engineers and Computer Scientists 2011 Vol I,IMECS 2011, March 16 - 18, 2011, Hong Kong
ISBN: 978-988-18210-3-4ISSN: 2078-0958 (Print); ISSN: 2078-0966 (Online)
IMECS 2011
 
This paper is structured as follows. In SectionII,first, we describe publicly available test collections, and second,elaborate on a test collection used in a specific research.SectionIIIdiscusses an online portal for semantic Webservices. Next, SectionIVidentifies Web crawlers that cu-mulate service descriptions, including our crawler. Finally,SectionVdistinguishes contests and challenges that use thesecollections. Related work that is presented in SectionVIIandSectionVIIIsummarizes the paper.II. A
VAILABLE
T
EST
C
OLLECTIONS
There are three existing major test collections, namely:SWS-TC, OWLS-TC, SAWSDL-TC. Additionally, there isanother unpublished test collection, which we call “100Services” throughout this paper.There exist some other collections, which are mentionedin [2], but are not included in our paper for reasons of unavailability, having non-standard formalisms, or havingformalisms other than the major ones available now.The four investigated test collections are differentiated andcompared in the following.
 A. SWS-T
SWS-TC
3
, generated manually by Ganjisaffar and Saboohiin 2006, contains 241 semantic services, mostly real Webservices. Services are described using OWL-S[3] description language. OWL-S version 1.1
4
is used in this test collection.SWS-TC contains 30 world-altering services. These serv-ice descriptions with preconditions and effects are writtenin SWRL [4]. Nevertheless, there is no expression availablein their precondition and effect specifications. Preconditionnames, effect names, and informal descriptions in naturallanguage are available, but formal SWRL specifications are“NIL.”There are two strong points for services available in SWS-TC. First, all the services are described with a single, unifiedontology derived from WordNet
5
. Therefore, the ontologydistance of concepts can be calculated for a matchmakingalgorithm. This is a unique feature in this test collection;others use multiple ontologies to describe the concepts. Thisis not a weak point for the others because, in the real world, itis somehow impossible to have a unified ontology for all theconcepts. Second, most of the services in this collection havebeen made from real Web services (they have been found inWeb service search engines), and they are not the so-called“toy” services.The limitations of SWS-TC are the following. First, thenumber of services is unsuitable for a test scenario. Second,the test collection is no longer being maintained and updated.
 B. OWLS-TC 
OWLS-TC
6
is an OWL-S service retrieval test collection.Services are described in OWL-S version 1.1 and some in
3
Semantic Web services’ test collection available athttp://www.semwebcentral.org/projects/sws-tc/ 
4
5
A lexical database for English, available athttp://wordnet.princeton.edu/ 
6
OWL-S Service Retrieval test collection available athttp://www.semwebcentral.org/projects/owls-tc/ 
OWL-S version 1.0
7
. This test collection contains serviceadvertisements, sample requests, and relevance judgmentsof the advertisements regarding to the requests. Klusch,Kapahnke et al. developed it in 2005 to support the eval-uation of the performance of OWL-S semantic Web servicematchmaking algorithms.The developers are actively improving OWLS-TC. Therewas no service, including precondition and effect, in thefirst three versions of this test collection. Some informalconditions written in services’ comments could not conveypractical world-altering service definitions.Recently, in September 2010, they have released the fourthversion of this collection. The latest release has servicecondition and effect specifications available both in SWRLand PDDL [5], which are bundled separately.This test collection contains 158 service descriptions pro-viding precondition and/or effect specifications. Among all,54 services are world-altering services. Furthermore, in thistest collection, there are five queries requiring world-alteringservices and 17 queries for services in need of preconditions.Comparing the number of world-altering service descriptionswith the total number of 1083 semantic services (5%) isaspirational, but not sufficient.
C. SAWSDL-TC 
A test collection similar to OWLS-TC is called SAWSDL-TC
8
. Klusch and Kapahnke also developed this test collec-tion. It was first converted from OWLS-TC. They made it tosupport the evaluation of the performance of SAWSDL [6]semantic Web service matchmaking algorithms.There is no condition or effect described in the formalismof this test collection. Therefore, the number of world-altering services in this test collection is unknown.
 D. 100 Services
In [7], researchers created their own test collection. This test collection seems to be the first in semantic Web servicestest collections, which contains the full description of 
world-altering
services. Prior to that, creators of SWS-TC describedsome world-altering services in their test collection. How-ever, as discussed in SectionII-A,these specifications of  preconditions and effects are not complete.This test collection is not publicly available. We couldanalyze it because it was graciously provided for us by thecreators. This allowed us to include it in our investigation.Services are described in OWL-S 1.1. They have usedmultiple ontologies for the concepts of inputs and outputsfor Web services.Among the 100 services available in this test collection,39 services are world-altering services. There are two uniquepreconditions and two unique results (effects). These precon-ditions and effects are in SWRL and replicated in variouscombinations for the services with assorted numbers andcombinations of inputs and outputs. This permutation gen-erated 39 distinctive world-altering services. Consequently,there are 61 information-providing services in this test col-lection.
7
8
SAWSDL Service Retrieval test collection available athttp://www.semwebcentral.org/projects/sawsdl-tc/ 
Proceedings of the International MultiConference of Engineers and Computer Scientists 2011 Vol I,IMECS 2011, March 16 - 18, 2011, Hong Kong
ISBN: 978-988-18210-3-4ISSN: 2078-0958 (Print); ISSN: 2078-0966 (Online)
IMECS 2011
 
III. A P
ORTAL FOR
S
EMANTIC
S
ERVICES
- OPOSS
UM
OPOSSum
9
is an online portal for semantic services [2]. It assembles data from SWS-TC, OWLS-TC and some othersources to create an assemblage of semantic Web serviceswith different description languages. It presently containsover 2800 descriptions for more than 1500 services.Unlike the test collections mentioned, OPOSSum is notfile-centered, and it is based on a relational database. Its webinterface enables the users to add, update, and improve thedata.Unfortunately, despite the OPOSSum developers’ hopethat the services and their descriptions would be improved bysemantic Web service community, the number of the servicesin this portal did not go further than 1600 in early 2011,which was almost equal to those in 2008.IV. W
EB
C
RAWLERS TO
C
UMULATE
An effort called Seekda’s Web Services portal
provides aWeb service search engine. It helps users find Web Servicesbased on a catalogue of more than 28,000 service descrip-tions. It utilizes a focused crawler that collects respectiveinformation about services available on the Web. The systemmonitors these services and allows users to edit certain dataregarding providers or services. Unfortunately, this portaldoes not support semantic services. As we have investigated,there is no facility in their search system to find world-altering services.There are some attempts to gather semantic service de-scriptions. These efforts try to use a Web crawler to look forpublic Web sites that publish semantic service descriptionsin one or more description languages.
 A. SouSuo
The meta-search engine SouSuo
1.6 searches for seman-tic Web services in semantic service description languageslike OWL-S, WSDL-S [8], WSMO [9], and SAWSDL, then evaluates the results in terms of their type, location, domain,and category [10].
 B. Our Crawler 
We have also configured a crawler (written in Java) to findservice descriptions publicly available on the web. The ob- jective was similar to the one in [10] with the distinguishingfeature that we were specifically looking for world-alteringservices. Likewise, the total number of semantic serviceswe could find was not satisfactory. Eventually, the numberof world-altering services, or even services with some kindof conditions, was small. Unfortunately, services containingSWRL definitions were few and far between.V. C
ONTESTS AND
C
HALLENGES
There are four major contests and challenges for variousresearch fields of semantic services. These include SWS-Challenge, S3 Contest, IEEE WSC, and SEALS SWS Disc-overy Evaluation.
9
10
11
 A. SWS
The SWS (Semantic Web Services) Challenge
intends todesign and develop a standard methodology and testbed forthe evaluation of Semantic Web Services technologies [11]. Currently, they focus on Web services described in WSDL2.0. The SWS Challenge is interested in comparing theeffectiveness of various formalisms for different problems.It has two tracks: data and process mediation, and servicediscovery.
 B. S3
The S3 (Semantic Service Selection) Contest
is a com-petition that measures the speed and precision of retrievalalgorithms in performing discovery tasks. Moreover, the con-test requires commitment to a specific semantic formalism.The S3 contest has different kinds of problem sets.The latest edition of the contest, which was in 2010, hadtwo tracks for OWL-S and SAWSDL matchmaker evalua-tions. The contest was conducted to compare the retrievalperformance for services in the OWL-S and the SAWSDLformats over test collections OWLS-TC and SAWSDL-TCrespectively.This contest uses SME
214
for evaluation purposes. SME
2
evaluates matchmakers for Semantic Web services over giventest collections in terms of standard retrieval performanceevaluation measures.
C. WS
The IEEE WSC (Web Services Challenge)
encouragesboth industry and academic researchers to participate. Theseinclude the groups that develop software components orintelligent agents. These applications should have the abilityto discover relevant web services and also generate compositeservices.The sixth competition, which was held in 2010, focusedexclusively on semantic composition of web service chains,whereas in the early editions, it was a syntactic-based contest.Rather than XML Schema, it incorporates the use of OWLontologies to define services and their relationships to eachother. The participants were required to determine relationsbetween different types during the process of service compo-sition.The IEEE WSC has its own test set generator. This toolgenerates an arbitrary number of services using any numberof concepts that the user likes. These concepts are alsorandomly generated and saved in an OWL taxonomy file.
 D. SEALS - SWS
The semantic Web services (SWS) track of the SEALSCampaign
is focused on the evaluation of algorithms andtools for semantic service discovery and matchmaking.
12
Semantic Web Service Challenge: Evaluating Semantic Web ServicesMediation, Choreography and Discovery, available athttp://www.sws-challenge.org/ 
13
Annual International Contest S3 on
Semantic Service Selection
- Re-trieval Performance Evaluation of Matchmakers for Semantic Web Services,http://www-ags.dfki.uni-sb.de/ 
14
The Semantic Web Service Matchmaker Evaluation Environment(SME
2
15
16
Proceedings of the International MultiConference of Engineers and Computer Scientists 2011 Vol I,IMECS 2011, March 16 - 18, 2011, Hong Kong
ISBN: 978-988-18210-3-4ISSN: 2078-0958 (Print); ISSN: 2078-0966 (Online)
IMECS 2011

You're Reading a Free Preview

Download
scribd
/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->