Social media platforms play an increasingly important civic role as platforms for discourse, where we discuss, debate, and share information. This article explores how users make sense of the content ...moderation systems social media platforms use to curate this discourse. Through a survey of users (n = 519) who have experienced content moderation, I explore users’ folk theories of how content moderation systems work, how they shape the affective relationship between users and platforms, and the steps users take to assert their agency by seeking redress. I find significant impacts of content moderation that go far beyond the questions of freedom of expression that have thus far dominated the debate. Raising questions about what content moderation systems are designed to accomplish, I conclude by conceptualizing an educational, rather than punitive, model for content moderation systems.
Social media companies make important decisions about what counts as “problematic” content and how they will remove it. Some choose to moderate hashtags, blocking the results for certain tag searches ...and issuing public service announcements (PSAs) when users search for troubling terms. The hashtag has thus become an indicator of where problematic content can be found, but this has produced limited understandings of how such content actually circulates. Using pro-eating disorder (pro-ED) communities as a case study, this article explores the practices of circumventing hashtag moderation in online pro-ED communities. It shows how (1) untagged pro-ED content can be found without using the hashtag as a search mechanism; (2) users are evading hashtag and other forms of platform policing, devising signals to identify themselves as “pro-ED”; and (3) platforms’ recommendation systems recirculate pro-ED content, revealing the limitations of hashtag logics in social media content moderation.
Most content moderation approaches in the United States rely on criminal justice models that sanction offenders via content removal or user bans. However, these models write the online harassment ...targets out of the justice-seeking process. Via an online survey with US participants (N = 573), this research draws from justice theories to investigate approaches for supporting targets of online harassment. We uncover preferences for banning offenders, removing content, and apologies, but aversion to mediation and adjusting targets’ audiences. Preferences vary by identities (e.g. transgender participants on average find more exposure to be undesirable; American Indian or Alaska Native participants on average find payment to be unfair) and by social media behaviors (e.g. Instagram users report payment as just and fair). Our results suggest that a one-size-fits-all approach will fail some users while privileging others. We propose a broader theoretical and empirical landscape for supporting online harassment targets.
Research suggests that marginalized social media users face disproportionate content moderation and removal. However, when content is removed or accounts suspended, the processes governing content ...moderation are largely invisible, making assessing content moderation bias difficult. To study this bias, we conducted a digital ethnography of marginalized users on Reddit’s /r/FTM subreddit and Twitch’s “Just Chatting” and “Pools, Hot Tubs, and Beaches” categories, observing content moderation visibility in real time. We found that on Reddit, a text-based platform, platform tools make content moderation practices invisible to users, but moderators make their practices visible through communication with users. Yet on Twitch, a live chat and streaming platform, content moderation practices are visible in channel live chats, “unban appeal” streams, and “back from my ban” streams. Our ethnography shows how content moderation visibility differs in important ways between social media platforms, harming those who must see offensive content, and at other times, allowing for increased platform accountability.
When evaluating automated systems, some users apply the “positive machine heuristic” (i.e. machines are more accurate and precise than humans), whereas others apply the “negative machine heuristic” ...(i.e. machines lack the ability to make nuanced subjective judgments), but we do not know much about the characteristics that predict whether a user would apply the positive or negative machine heuristic. We conducted a study in the context of content moderation and discovered that individual differences relating to trust in humans, fear of artificial intelligence (AI), power usage, and political ideology can predict whether a user will invoke the positive or negative machine heuristic. For example, users who distrust other humans tend to be more positive toward machines. Our findings advance theoretical understanding of user responses to AI systems for content moderation and hold practical implications for the design of interfaces to appeal to users who are differentially predisposed toward trusting machines over humans.
Public debate about content moderation has overwhelmingly focused on removal: social media platforms deleting content and suspending users, or opting not to do so. However, removal is not the only ...available remedy. Reducing the visibility of problematic content is becoming a commonplace element of platform governance. Platforms use machine learning classifiers to identify content they judge misleading enough, risky enough, or offensive enough that, while it does not warrant removal according to the site guidelines, warrants demoting them in algorithmic rankings and recommendations. In this essay, I document this shift and explain how reduction works. I then raise questions about what it means to use recommendation as a means of content moderation.
In August 2019, a mass shooter in the United States posted a violent manifesto to the anonymous forum 8chan prior to his attack. This was the third such incident that year and afterwards hosting and ...security services conceded to calls to drop 8chan as a client, pushing 8chan to the margins of the accessible internet. This article examines the deplatforming of 8chan as a public relations crisis, contributing to understanding ‘governance by shock’ (Ananny and Gillespie 2016) by examining who is shocked and their power to turn shock into online regulation. Online platforms and media attention created opportunities to study how the deplatforming was justified, drawing on the theoretical framework of economies of worth (Boltanski and Thevenot 2006) and controversy mapping methods. The examination finds: (1) that this case of deplatforming indicates the openness of infrastructure-as-a-service companies to external challenges over content, rather than hegemonic control. (2) That regulatory gaps, including the broadness of U.S. free speech laws, made these companies, rather than legal processes, the relevant authority. (3) That framing responsibility as following the law – as Cloudflare attempted to do – misunderstands the importance of normative principles, voluntary measures, and contestation in governing online content, underselling the value of policy-making at other levels. The success of the campaign to deplatform 8chan affirms the significance of PR crises in the regulation of online content, rewarding deplatforming as a political tactic for civil society groups and online networks pushing for governance in regulatory gaps. However, the significance of normative enforcement in this case underlines the difficulties of this semi-voluntary style of governance. While normative opposition to violence contributed to 8chan’s deplatforming, other normative oppositions contribute to deplatforming vulnerable users, as in the moral panics that drive the deplatforming of sexual content ( Tiidenberg 2021 ) and feed suspicion over the ideological application of deplatforming. The ambivalence of PR crises as a strategy for influencing platform governance underlines the need for clarity in policy-making at multiple levels.
Content regulation on digital platforms has become a contested issue on the public and scholarly agendas. To understand how digital platform providers experiment with making commitments regarding ...their regulation, this article process-traces Facebook’s content regulation to ask how it self-regulates despite constant pressures for policy intervention. The first part of the article shows how Facebook moved from its initial “thin” self-regulatory regime toward what I call “enhanced self-regulation,” which relies on first-party and independent third-party intermediaries. Thereafter, I show how Facebook self-regulated the balance between public and private interests over time and across the regimes. The findings suggest that powerful actors such as Facebook can innovate in self-regulation by reallocating content-related responsibilities to intermediaries and subsequently create polycentric governance regimes. Lessons about how self-regulators that face public criticism can make more credible commitments to public interests are then drawn from the strengths and weakness of enhanced self-regulation.
This study examines social media users’ preferences for the use of platform-wide moderation in comparison to user-controlled, personalized moderation tools to regulate three categories of ...norm-violating content—hate speech, sexually explicit content, and violent content. Via a nationally representative survey of 984 US adults, we explore the influence of third-person effects and support for freedom of expression on this choice. We find that perceived negative effects on others negatively predict while free speech support positively predicts a preference for having personal moderation settings over platform-directed moderation for regulating each speech category. Our findings show that platform governance initiatives need to account for both actual and perceived media effects of norm-violating speech categories to increase user satisfaction. Our analysis also suggests that users do not view personal moderation tools as an infringement on others’ free speech but as a means to assert greater agency over their social media feeds.
This article proposes ‘sexist assemblages’ as a way of understanding how the human and mechanical elements that make up social media content moderation assemble to perpetuate normative gender roles, ...particularly white femininities, and to police content related to women and their bodies. It investigates sexist assemblages through three of many potential elements: (1) the normatively gendered content presented to users through in-platform keyword and hashtag searches; (2) social media platforms’ community guidelines, which lay out platforms’ codes of conduct and reveal biases and subjectivities and (3) the over-simplification of gender identities that is necessary to algorithmically recommend content to users as they move through platforms. By the time the reader finds this article, the elements of the assemblages we identify might have shifted, but we hope the framework remains useful for those aiming to understand the relationship between content moderation and long-standing forms of inequality.
PDF
