Choice overload and recommendation effectiveness in related-article recommendations

Choice overload describes a situation in which a person has difficulty in making decisions due to too many options. We examine choice overload when displaying related-article recommendations in digital libraries, and examine the effectiveness of recommendation algorithms in this domain. We first analyzed existing digital libraries, and found that only 30% of digital libraries show related-article recommendations to their users. Of these libraries, the majority (74%) displays 3–5 related articles; 28% of them display 6–10 related articles; and no digital library displayed more than ten related-article recommendations. We then conducted our experimental evaluation through GESIS’ digital library Sowiport, with recommendations delivered by recommendations-as-a-service provider Mr. DLib. We use four metrics to analyze 41.3 million delivered recommendations: click-through rate (CTR), percentage of clicked recommendation sets (clicked set rate, CSR), average clicks per clicked recommendation set (ACCS), and time to first click (TTFC), which is the time between delivery of a set of recommendations to the first click. These metrics help us to analyze choice overload and can yield evidence for finding the ideal number of recommendations to display. We found that with increasing recommendation set size, i.e., the numbers of displayed recommendations, CTR decreases from 0.41% for one recommendation to 0.09% for 15 recommendations. Most recommendation sets only receive one click. ACCS increases with set size, but increases more slowly for six recommendations and more. When displaying 15 recommendations, the average clicks per set is at a maximum (1.15). Similarly, TTFC increases with larger recommendation set size, but increases more slowly for sets of more than five recommendations. While CTR and CSR do not indicate choice overload, ACCS and TTFC point toward 5–6 recommendations as being optimal for Sowiport. Content-based filtering yields the highest CTR with 0.118%, while stereotype recommendations yield the highest ACCS (1.28). Stereotype recommendations also yield the highest TTFC. This means that users take more time before clicking stereotype recommendations when compared to recommendations based on other algorithms.