Design of CQA Systems for Flexible and Scalable Deployment and Evaluation

Successfulness of Community Question Answering (CQA) systems on the open web (e.g. Yahoo! Answers) motivated for their utilization in new contexts (e.g. education or enterprise) and environments (e.g. inside organizations). In spite of initial research how their specifics influence design of CQA systems, many additional problems have not been addressed so far. Especially a poor flexibility and scalability which hamper: (1) CQA essential features to be employed in various settings (e.g. in different educational organizations); and (2) collaboration support methods to be effectively evaluated (e.g. in offline as well as in live experiments). In this paper, we provide design recommendations how to achieve flexible and scalable deployment and evaluation by means of a case study on educational and organizational CQA system Askalot. Its universal and configurable features allow us to deploy it at two universities as well as in MOOC system edX. In addition, by means of its experimental infrastructure, we can integrate various collaboration support methods which are loosely coupled and can be easily evaluated online as well as offline with datasets from Askalot itself or even from all CQA systems built on the top of the Stack Exchange platform.