A system for ranking organizations using social scale analysis

In this paper, we utilize feature extraction and model-fitting techniques to process the rhetoric found in the web sites of 23 Indonesian Islamic religious organizations to profile their ideology and activity patterns along a hypothesized radical/counter-radical scale, and present an end-to-end system that is able to help researchers to visualize the data in an interactive fashion on a timeline. The subject data of this study is 37,000 articles downloaded from the web sites of these organizations dating from 2001 to 2011. We develop algorithms to rank these organizations by assigning them to probable positions on the scale. We show that the developed Rasch model fits the data using Andersen’s LR-test. We create a gold standard of the ranking of these organizations through an expertise elicitation tool. We compute expert-to-expert agreements, and we present experimental results comparing the performance of three baseline methods to show that the Rasch model not only outperforms the baseline methods, but it is also the only system that performs at expert-level accuracy.