11. Evaluation of Structured Collaborative Tagging for Web Service Matchmaking

Turning Web services into Semantic Web Services (SWS) can be prohibitively expensive for large repositories. In such cases collecting community descriptions in forms of structured tags can be a more affordable approach to describe Web services. However, little is understood about how tagging impacts performance of retrieval. There are neither real structured tagging systems for Web services, nor real corpora of structured tags. To start addressing these issues, in our approach, we motivated taggers to tag services in a partially controlled environment. Specifically, taggers were given application requirements and asked to find and tag services that match the requirements. Tags collected in this way were used for Web service matchmaking and evaluated within the framework of the Cross-Evaluation Track of the Third Semantic Service Selection 2009 contest. As part of the lessons learned, we explain relations between description schema (SWS, tags, flat document) and matchmaking heuristics, and performance of retrieval in different search scenarios. We also analyze reliability of tagging system performance as related to taggers/searchers autonomy. Finally, we identify threats to results credibility stemming from partial control of the tags collection process.