Disinformation in Open Online Media

In times of massive fake news campaigns in social media, one may ask who is to blame for the spread of misinformation online. Are humans, in their limited capacity for rational self-reflection or responsible information use, guilty because they are the ones falling for the misinformation? Or are algorithms that provide the basis for filter bubble phenomena the cause of the rise of misinformation in particular in the political public discourse? In this paper, we look at both perspectives and see how both sides contribute to the problem of misinformation and how underlying metrics shape the problem.

In addition to the increased opportunities for citizens to participate in society, participative online journalistic platforms offer opportunities for the dissemination of online propaganda through fake accounts and social bots. Community managers are expected to separate real expressions of opinion from manipulated statements through fake accounts and social bots. However, little is known about the criteria by which managers make the distinction between “real” and “fake” users. The present study addresses this gap with a series of expert interviews. The results show that community managers have widespread experience with fake accounts, but they have difficulty assessing the degree of automation. The criteria by which an account is classified as “fake” can be described along a micro-meso-macro structure, whereby recourse to indicators at the macro level is barely widespread, but is instead partly stereotyped, where impression-forming processes at the micro and meso levels predominate. We discuss the results with a view to possible long-term consequences for collective participation.

The endemic spread of misinformation online has become a subject of study for many academic disciplines. Part of the emerging literature on this topic has shown that conspiracy theories (CTs) are closely related to this phenomenon. One of the strategies deployed to combat this online misinformation is confronting users with corrective information, often drawn from mainstream media outlets. This study tries to answer the questions (I) whether there are online-communities that exclusively consume conspiracy theorist media and (II) how these communities use information sources from the mainstream. The results of our explorative, large-scale content analysis show that even in conspiracy theorist communities, mainstream media sources are being used very similar to sources from the conspiracy theorist media spectrum, thus not reaching any of their assumed corrective potential.

In online media environments, nostalgia can be used as important ingredient of propaganda strategies, specifically, by creating societal pessimism. This work addresses the automated detection of nostalgic text as a first step towards automatically identifying nostalgia-based manipulation strategies. We compare the performance of standard machine learning approaches on this challenge and demonstrate the successful transfer of the best performing approach to real-world nostalgia detection in a case study.

Abusive language has been corrupting online conversations since the inception of the internet. Substantial research efforts have been put into the investigation and algorithmic resolution of the problem. Different aspects such as “cyberbullying”, “hate speech” or “profanity” have undergone ample amounts of investigation, however, often using inconsistent vocabulary such as “offensive language” or “harassment”. This led to a state of confusion within the research community. The inconsistency can be considered an inhibitor for the domain: It increases the risk of unintentional redundant work and leads to undifferentiated and thus hard to use and justifiable machine learning classifiers. To remedy this effect, this paper introduces a novel configurable, multi-view approach to define abusive language concepts.

Since the Brexit vote and the 2016 U.S. election, much has been speculated about the use of so-called social bots, (semi-)automatized pseudo-users in online media, as political manipulation tools. Accumulating global evidence shows that pseudo-users are used for different purposes, such as the amplification of political topics or the simulation of large numbers of followers. Social bots, as a (semi-)automated pseudo-user type, are part of a larger infrastructure, among others, entailing network access, fake accounts, and hosting services. Users and providers of social bots and their infrastructure can differ. Thus, it is plausible that a digital goods market has emerged for the exchange of social bots and infrastructure components. The present study used an ethnographic approach to study the accessibility, availability, and prices for pseudo-users and social bots on markets in the (German- and English-language) Clearnet and Darknet. The results show that an infrastructure for digital manipulation is widely available online, and that the tools for artificial content or connectedness amplification are easily accessible for lay users and are cheap on Clearnet and Darknet markets.

Social bots have recently gained attention in the context of public opinion manipulation on social media platforms. While a lot of research effort has been put into the classification and detection of such automated programs, it is still unclear how technically sophisticated those bots are, which platforms they target, and where they originate from. To answer these questions, we gathered repository data from open source collaboration platforms to identify the status-quo of social bot development as well as first insights into the overall skills of publicly available bot code.

In this work we investigate the engagement of Twitter accounts in the starting phase of reaction cascades, i.e., in the follow-up stream of an original tweet. In a first case study, we focus on a selection of very popular Twitter users from politics and society. We find a small but constantly active set of seemingly automated accounts in the onset of cascades that may contribute to the multiplication of content–especially for well-known populist politicians.

Online platforms providing information and media content follow certain goals and optimize for certain metrics when deploying automated decision making systems to recommend pieces of content from the vast amount of media items uploaded to or indexed by their platforms every day. These optimization metrics differ markedly from, for example, the so-called news factors journalists traditionally use to make editorial decisions. Social networks, video platforms and search engines thus create content hierarchies that reflect not only user interest but also their own monetization goals. This sometimes has unintended, societally highly problematic effects: Optimizing for metrics like dwell time, watch time or “engagement” can promote disinformation and propaganda content. This chapter provides examples and discusses relevant mechanisms and interactions.

Alternative news media with a right-wing populist leaning are flourishing. They pitch themselves as opposition to a hegemonically interpreted mainstream news media system. Yet, at the same time, they rely on the so criticized others to justify their own existence. Using a co-orientation framework, the current study asked in how far right-wing populist alternative news media orient themselves towards the mainstream. Using a qualitative content analysis of all 658 websites referenced by a popular right-wing conspiracy-theoretical YouTuber in Germany, we demonstrate that distinct source types were quoted. References ranged from mainstream news media up to ultra-right wing truther blogs. A quantitative examination of the content-analytical categories confirmed significant differences between mainstream news media and right-wing populist blogs, with special interest and alternative news media ranging in between these poles. Alternative news media were overall found to orient themselves stylistically strongly towards the mainstream but less so regarding their content selection. Particularly, the top sources, accounting for over 76% of all references, were mostly rooted in the alternative ultra-right-wing ecosystem. In sum, our analyses showed how stylistic co-orientation is used to build a bridge towards the mainstream while content-related co-orientation towards other ultra-right-wing alternative sources allows for validating one’s own right-wing populist worldview.

In recent years, a variety of studies has discussed the use of social media in the context of misinformation, fake news and manipulation of public opinion. Based on two data sets published by Twitter, including more than 1.7 million English-language tweets, this study focuses on the question whether Iranian propaganda accounts tried to influence the international online debate on the country’s biggest rival, Saudi Arabia. The rivalry between both countries has been an ongoing fight deeply rooted in a regional, geopolitical, ideological and somewhat religious conflict. An analysis of the tweets published by the accounts which are believed to be connected to Iranian state-backed information operations has shown that they have tried to establish an anti-Saudi narrative on Twitter. Different strategies, including the spread of biased hashtags or retweeting internal and external propaganda sources, were used to promote their agenda. The propaganda activity on Saudi Arabia was especially distinctive during specific time intervals, correlating with political events, but has regularly failed to manipulate the international discourse. Although some content that negatively mentioned Saudi Arabia was actively retweeted, the vast majority did not influence the social media debate on the Gulf state.