CSQuaRE: Approach for Quality Control in Crowdsourcing

Quality control of responses enhances sustainability and adoption of crowdsourcing. Expert and peer reviews, majority voting, machine learning, game theory, etc. are some of the practices for quality control in crowdsourcing. However, quality of crowdsourced responses is still a concern. We propose a quality control approach drawing inspiration from Requirements Engineering quality attributes - Completeness, Consistency and Correctness (3Cs). The 3Cs of a response are assessed and displayed as a CSQuaRE score based on coverage with reference to knowledge base (ontology), cohesiveness and contributor credibility in the domain. The knowledge base would evolve with continued extraction of instances of information and thus responses would be re-calibrated for relevance. The suggested approach would be demonstrated for Information Security related Question and Answers on a crowdsourcing platform. The evaluation of the approach would be based on comparison with existing quality control techniques and feedback from security experts.