The Evolution of Trust and Safety

Danielle Keats Citron* & Ari Ezra Waldman**

This Article tells the inside story of the rise, spread, and evolution of trust and safety. We bring
unique perspectives to this story. One of us (Citron) was on the inside, advising many tech
companies over the years on how to create healthy digital environments. The other (Waldman) was
on the outside, observing, through field research, how tech companies respond to and translate law
into their routines, organizational structures, and practices. Together, these perspectives offer
scholars and policymakers unmatched opportunities not just to see what is happening, but to
understand it as well. For decades, U.S. tech companies resisted accountability for user-generated
abuse by pointing to their content moderation practices, safety features, and the teams behind them.
By the mid-2010s, the biggest social media companies had hundreds of employees (hailing from
law, social science, engineering, law enforcement, and other backgrounds) and thousands of
contractors working on what came to be known as “trust and safety”. Trust and safety evolved into
a field with its own conferences, journal, and 2,000 plus member professional organization. Of late,
however, tech companies have been firing trust and safety personnel, tolerating or worse welcoming
more abuse, and removing safety features. The trust and safety field appears to be collapsing or
disappearing. But appearances are deceiving. Trust and safety efforts are not collapsing but instead
evolving to focus on more on compliance than a holistic approach aimed to secure healthy digital
environments. The character of trust and safety is changing. Collaborations pushing for tech safety
and accountability have been replaced by paperwork.

INTRODUCTION
Social media started small. For much of 2005, Facebook was only available to students
at a few elite colleges. 1 As social media platforms like Facebook grew in popularity, more
anti-social activity was hosted on those sites. 2 As more people suffered harm, more
people objected. 3 Investigative reporters documented harms; 4 advocates called for

* Jefferson Scholars Foundation Schenck Distinguished Professor of Law, Martha Lubin Karsh and Bruce
A. Karsh Bicentennial Professor of Law, University of Virginia School of Law; Co-Director, UVA LawTech
Center; Vice President and Secretary, Cyber Civil Rights Initiative; 2019 MacArthur Fellow.
** Professor of Law and, by courtesy, Professor of Sociology, University of California Irvine School of Law;

Founding Director, Center for Law, Technology, & Society, UC Irvine School of Law; Board of Directors,
Cyber Civil Rights Initiative. We thank David Del Terzo, Charlotte Karlsen, Adeleine Lee, Laurel Lehman,
Jonathan McCutchen, Jeff Stautberg, and Alexander Wilfert for superb research assistance and Kate
Boudoris for as-ever superb advice and research guidance. We are grateful to Isabella Ryan and Julia
Bettencourt for including us in the Emory Law Journal symposium. The authors contributed equally to this
article, mistakes and all, so our names are listed alphabetically.
1 TARLETON GILLESPIE, CUSTODIANS OF THE INTERNET: PLATFORMS, CONTENT MODERATION, AND THE HIDDEN

DECISIONS THAT SHAPE SOCIAL MEDIA 118 (2018).

2 See DANIELLE KEATS CITRON, HATE CRIMES IN CYBERSPACE 3-5 (2014); DANIELLE KEATS CITRON, THE FIGHT

FOR PRIVACY: PROTECTING DIGNITY, IDENTITY, AND LOVE IN THE DIGITAL AGE 24-33 (2022); Danielle Keats
Citron, Sexual Privacy, 128 YALE L.J. 1870, 1908-23 (2018).
3 See CITRON, FIGHT FOR PRIVACY, supra note 2, at 190-96; Mary Anne Franks, “Revenge Porn” Reform: A View

from the Front Lines, 69 FLA. L. REV. 1251, 1278-92 (2017); Citron, Sexual Privacy, supra note 2, at 1944-45.
4 The extraordinary journalists highlighting destructive abuse included: Emily Bazelon, Why Do We

Tolerate Revenge Porn?, SLATE (Sept. 25, 2013), https://slate.com/human-interest/2013/09/revenge-

1
change; 5 policymakers expressed concern; 6 firms threatened to pull their
advertisements. 7 These developments helped create the field now known as digital trust
and safety. This Article tells the inside story of trust and safety’s evolution, highlighting
the catalyzing role that law has played along the way.

With fits and starts, social media and other content platforms developed initiatives to
tackle online abuse. 8 It started ad hoc and “artisanal”—for instance, an employee of a
social media company would receive abuse complaints and decide that one particular
video would remain online whereas another video would be removed. 9 As some
platforms grew, individual employees wrote speech policies prohibiting certain speech
activity, such as fraud, spam, nudity, bullying, hate speech, and nonconsensual intimate
imagery, which in-house workers and contractors enforced. 10 A decade into social
media’s expansion, companies began thinking about safety-by-design and forged

porn-legislation-a-new-bill-in-california-doesnt-go-far-enough.html; Jessica Guynn & Janet Stobart,

Ask.fm: New Social Site, Same Bullying, L.A. TIMES (Aug. 20, 2013),
https://www.latimes.com/business/la-xpm-2013-aug-20-la-fi-britain-cyber-bullying-20130820-
story.html; Kashmir Hill, Hunter Moore Will Post Your Nude Photos But Only If He Thinks You Are A
Horrible Person, FORBES.COM, December 5, 2012,
http://www.forbes.com/sites/kashmirhill/2012/12/05/hunter-moore-is-going-to-start-posting-
your-nude-photos-again-but-will-only-post-your-home-address-if-he-thinks-youre-a-horrible-
person/; Jessica Roy, Hunter Moore’s Scary as Shit Revenge Porn Site Will Map Submitted Photos to People’s
Addresses, OBSERVER.COM (Nov. 27, 2012), http://betabeat.com/2012/11/hunter-moores-scary-as-shit-
revenge-porn-site-will-map-submitted-photos-to-peoples-addresses/; Jessica Bennett, Should Facebook
Ban Sexist Pages?, THE DAILY BEAST (Nov. 5, 2011).
5 https://cybercivilrights.org/ccri-history/; Mary Anne Franks, “Revenge Porn” Reform: A View from the

Front Lines, 69 FLA. L. REV. 1251, 1278-92 (2017); Citron, Sexual Privacy, supra note 2, at 1944-45; Danielle
Citron & Mary Anne Franks, Evaluating New York’s “Revenge Porn” Law: A Missed Opportunity to Protect
Sexual Privacy, HARV. L. REV. BLOG (Mar. 19, 2019),
https://harvardlawreview.org/blog/2019/03/evaluating-new-yorks-revenge-porn-law-a-missed-
opportunity-to-protect-sexual-privacy/; Ari Waldman, For Tyler: How One Man Is Taking on Cyberbullies,
ADVOCATE (Oct. 8, 2015), https://www.advocate.com/commentary/2015/10/08/tyler-how-one-man-
taking-cyberbullies.
6 AG Harris Debuts New Hub of Resources for Law Enforcement, Tech, and Victims of Cyber Exploitation,

YOUTUBE (Oct. 15, 2015), https://www.youtube.com/watch?v=clLB8frv568; Patrick May, California’s

Attorney General Launches a War on ‘Revenge Porn’ and Other Cyber Crimes, SILICON VALLEY.COM (Oct. 14,
2015), https://www.siliconvalley.com/2015/10/14/californias-attorney-general-launches-a-war-on-
revenge-porn-and-other-cyber-crimes/; Danielle Citron, Attorney General Kamala Harris to Help Law
Enforcement in Investigations of Criminal Invasions of Sexual Privacy, IACP,
https://www.iacpcybercenter.org/the-groundbreaking-work-of-attorney-general-kamala-harris-to-help-
law-enforcement-in-investigations-of-criminal-invasions-of-sexual-privacy/.
7 CITRON, HATE CRIMES IN CYBERSPACE, supra note 2, at 229-30.
8 Id. at 227-36.
9 Robyn Caplan, Content or Context Moderation: Artisanal, Community-Reliant, and Industrial Approaches, at 1,

Data & Soc’y (Nov. 14, 2018), https://datasociety.net/library/content-or-context-moderation/.

10 See Kate Klonick, The New Governors: The People, Rules, and Processes Governing Online Speech, 131 HARV.

L. REV. 1598, 1620 (2018); Kate Klonick, The Facebook Oversight Board: Creating an Independent Institution to
Adjudicate Online Free Expression, 129 YALE L.J. 2232 (2020); Ari Ezra Waldman, Disorderly Content, 97 WASH.
L. REV. 907 (2022).

2
collaborations with civil society and academics to share expertise and enhance
transparency. 11 The biggest companies built trust and safety teams; employees moved
around to other companies; platforms copied each other.

People working on these efforts had varied backgrounds; they were lawyers, social
scientists, engineers, policy experts, and former law enforcement agents. 12 They came to
describe their work as trust and safety—meaning that their purpose was to “maintain the
integrity of online platforms and digital spaces,” combat fraud, abuse, harassment, and
cyber threats, and “foster an environment of trust among users.” 13 In 2020, the Trust and
Safety Professional Association (TSPA) was established to support and educate those
working in the field. 14 The TSPA and the Stanford Cyber Policy Center established a trade
journal, the Journal of Online Trust and Safety, and have jointly run trust and safety
conferences in the United States and across the globe. 15 Thousands attend these
conferences every year. 16

Despite this unprecedented and rapid growth of trust and safety, scholars and
commentators have forecasted its demise. In 2023, Kate Klonick, a law professor, and
Casey Newton, a technology journalist, endorsed this view. For Klonick, the “golden age
of tech accountability” was over;17 for Newton, trust and safety work was past its
“peak.” 18 Other scholars have called it the “end” of trust and safety given the economic
contraction in the tech industry. 19 They were responding to high-profile examples of
falling trust-and-safety dominos: After Elon Musk bought Twitter (now X), he fired most
of the company’s trust and safety employees, removed safety features, welcomed back

11 Evelyn Douek, Content Moderation as Systems Thinking, 136 HARV. L. REV. 526 (2022). Trust and safety
professionals now perform many tasks inside an organization. See Toby Shulruff, Trust and Safety Work:
Internal Governance of Technology Risks and Harms, 1 J. INTEGRATED GLOB. STUD. 95, 98-100 (2024) (identifying
six overlapping functions and necessary skills).
12 Careers in Trust & Safety, https://www.tspa.org/careers/ (last visited Aug. 15, 2025).
13 What is Trust & Safety?, https://alltechishuman.org/trust-and-safety (last visited Aug. 15, 2025).
14 Key Elements of a Trust and Safety Team, Trust and Safety Professional Association,

https://www.tspa.org/curriculum/ts-fundamentals/industry-overview/ts-team-elements/ (last visited

Aug. 15, 2025).
15 See, e.g., The 4th Annual Trust and Safety Research Conference, Stanford Cyber Policy Center,

https://cyber.fsi.stanford.edu/content/trust-and-safety-research-conference-2025 (last visited Aug. 15,

2025); TrustCon2025, https://www.trustcon.net/event/0f932abb-fad0-4e42-a1b7-7563f6123e41/summary
(last visited Aug. 15, 2025).
16 TrustCon 2025 featured 303 speakers and nearly 2,000 attendees. See Speakers, TrustCon,

https://www.trustcon.net/event/0f932abb-fad0-4e42-a1b7-7563f6123e41/speakers (last visited July 29,

2025).
17 Kate Klonick, The End of the Golden Age of Tech Accountability, SUBSTACK (Mar. 3, 2023),

https://klonick.substack.com/p/the-end-of-the-golden-age-of-tech.
18 Casey Newton, Have We Reached Peak Trust and Safety?, SUBSTACK (June 8, 2023).
https://platformer.substack.com/p/have-we-reached-peak-trust-and-safety.
19 Rachel Elizbeth Moran, Joseph Schafer, Mery Bayar, & Kate Starbird, The End of Trust and Safety?

Examining the Future of Content Moderation and Upheavals in Professional Online Safety Efforts, at 4, CHI
Conference on Hum.-Factors in Comput. Sys. (2025), doi: 10.1145/3706598.3713662.

3
banned accounts, and substituted content moderation with user feedback. 20 Then came
the cascade: Meta, YouTube, and Twitch fired trust and safety staff and cut back on their
efforts. 21

Klonick, Newton, and others observed these developments and made two related
arguments: First, trust and safety was collapsing, and second, the collapse was due to
financial pressure. Klonick described the firings as a “blood-letting” and the “loss” of
entire programs; 22 Newton quoted David Thiel, chief technologist at the Stanford Internet
Observatory and former Facebook employee, who stated that trust and safety was being
“decimated” and “under attack.” 23 Both attributed trust and safety’s alleged fall to
corporate budgetary constraints. Klonick underscored that the contraction was occurring
during an “economic downturn”. 24 “[M]assive revenues,” Klonick argued, gave
platforms breathing space to curb abuse, “[b]ut when the money stops, the first projects
to lose funding are” self-regulatory efforts like trust and safety. 25 Citing Klonick, Newton
noted that “the plunge in tech company valuations over the past years seems to have
reduced enthusiasm” for trust and safety initiatives. 26 Trust and safety efforts were first
on the chopping block when executives had to make cuts.

These arguments are incomplete. Perceptions of “peak” or “golden ages” of trust and
safety embed static conceptions of what the field is and should be; economic narratives
of its fall are convenient but elide the economic pressures fomenting trust and safety’s
rise and the sociolegal forces that made trust and safety precarious from the start. The
field of trust and safety is not declining but instead evolving, not dying but rather
changing shape. The change is not simply due to economics, but also to new laws and
practices of the individuals and organizational forms that make up the field.

The rise, diffusion, and evolution of trust and safety is a sociolegal story, not a purely
economic one. It is a case study in the ground-up development of law and practice, 27 the
ways in which those practices spread among organizations isomorphically, 28 and the role
of law triggering both.

20 See Part II.

21 See Part II.
22 Klonick, supra note 17.
23 Newton, supra note 18.
24 Klonick, supra note 17.
25 Id.
26 Newton, supra note 18 (also noting that “overall investment in trust and safety teams has declined …

amid mass layoffs in the tech industry”).

27 See Lauren Edelman, Christopher Uggen, & Howard Erlanger, The Endogeneity of Legal Regulation:

Grievance Procedures as Rational Myth, 105 AM. J. SOCIOLOGY 406 (1999); Ari Ezra Waldman, Privacy Law’s
False Promise, 97 WASH. U. L. REV. 773 (2019).
28 See Paul J. DiMaggio & Walter W. Powell, The Iron Cage Revisited: Institutional Isomorphism and Collective

Rationality in Organizational Fields, 48 AM. SOCIOLOGICAL REV. 147 (1983); see also John W. Meyer & Brian
Rowan, Institutionalized Organizations: Formal Structure as Myth and Ceremony, 83 AM. J. SOCIO. 340, 340

4
 This Article proceeds in three parts. Part I asks: How did the field of trust and safety
begin and spread across the information industry? We argue that the passage of Section
230 of the Communications Decency Act, 29 which invested platforms with power to
moderate content in exchange for legal immunity related to user-developed content,
catalyzed the birth and diffusion of trust and safety practices. By creating a system of self-
governance for platforms, Section 230 ushered in freedom and in turn significant
uncertainty among tech companies as to the best way to address harmful content. This
uncertainty triggered two developments—the endogenous development of systems,
routines, and internal corporate practices falling under the umbrella of trust and safety
and mimicry among competitors in the same field, or a version of what Paul DiMaggio
and Walter Powell called “institutional isomorphism.” 30 The resulting systems were far
from perfect; we have been vocal critics of trust and safety self-governance. 31 This Part
puts that criticism aside and focuses on describing trust and safety from an institutional
perspective.

Part II asks: Is trust and safety collapsing? We argue that it is not. The tenor and center
of gravity of the profession, not to mention the form of its practices, are changing. This
evolution has law to thank for it, too: New laws and enforcement investigations in the
United Kingdom and the European Union have triggered a shift from self-governance to
a public-private partnership involving record-keeping, transparency reports, outside
vendors, proceduralism, and the automation of compliance.

Part III asks: What is the future of trust and safety? The field’s turn to compliance is
not necessarily a good thing. This Part explores the potential harms of this shift. Retreat
from holistic trust and safety work, as insufficient as it may have been, may sacrifice
valuable collaborative efforts that increased industry knowledge and enabled advocates
to push for greater trust and safety efforts. Harms may go unaddressed if companies
replace action with paperwork. As trust and safety continues to evolve, the pieces we
have lost may be irretrievable. The reconceptualization of trust and safety as a compliance
task and the resulting over-reliance on not-so-intelligent bots and automation may make
us less inclined in the future to spend money to hire human beings and to learn from the
creativity and imagination that they bring.

(1977). This Article focuses on the rise, evolution, and isomorphic diffusion of trust and safety practices,
relying on neo-institutional theory in the sociology of organizations. We will focus on trust and safety
boundary work, the processes by which professional actors define the jurisdictional limits of their field, in
future scholarship. See ANDREW ABBOTT, THE SYSTEM OF PROFESSIONS (1988); Thomas F. Gieryn, Boundary-
Work and the Demarcation of Science from Non-Science: Strains and Interests in Professional Ideologies of Scientists,
48 AM. SOCIOLOGICAL REV. 781 (1983).
29 47 U.S.C. § 230.
30 DiMaggio & Powell, supra note 28, at 148.
31 See, e.g., Waldman, Disorderly Content, supra note 10; CITRON, FIGHT FOR PRIVACY, supra note 2, at 25-38.

5
 This Article makes several important and novel contributions to our understanding
of trust and safety. We bring together insider and outsider perspectives on the growth of
the field. 32 One of us (Citron) began advising tech companies in 2009, continuing that
work until 2024; 33 the other (Waldman) has conducted several years of fieldwork
observing and interviewing tech company workers and reviewing their internal
documents to understand how they translate law on the books into law on the ground.34
Together, we interviewed trust and safety professionals for background information. We
apply theories and methodologies in sociological theory and organizational sociology to
trust and safety, enabling us to see the limitations of the conventional wisdom of trust
and safety’s rise and supposed collapse due to economic pressures. We highlight the
evolution of trust and safety efforts, take stock of the consequences, and emphasize the
role of law. We reveal both the fragility of self-regulation and the consequences of a shift
to compliance before suggesting how changes in law can trigger further evolution of trust
and safety into something new and more effective than before.

I. TRUST AND SAFETY’S DEVELOPMENT

This Part traces the rise and diffusion of trust and safety practices in the United States.
It argues that the law—Section 230 in particular—created a self-governance regime with
significant legal uncertainty for platforms. To reduce that uncertainty, leading platforms
developed and then formalized trust and safety practices endogenously, or from the
ground-up. Competitors learned and copied these strategies, spreading those practices
throughout the field.

A. The Section 230 Trigger

Law plays an active role in shaping market dynamics. 35 Focusing on the rise of the
information industry writ large, Julie Cohen challenged the conventional assumption

32 The few academic studies of trust and safety are drawn from interviews with insiders at tech companies.
See, e.g., Megan Knittel & Amanda Menking, Bridging Theory and Practice: Examining the State of Scholarship
Using the History of Trust and Safety Archive, 2 J. ONLINE TRUST & SAFETY 1 (2024),
https://tsjournal.org/index.php/jots/article/view/173/63?ref=everythinginmoderation.co; Lucas
Wright, The Salesforce of Safety: Software Vendors as Infrastructural/Professional Nodes in the Field of Online Trust
and Safety (under submission) (manuscript on file with authors).
33 Citron advised Twitter, Facebook, TikTok, Twitch, Snap, Bumble, Google’s Jigsaw, Microsoft, and

Spotify. Most of that work was unpaid and on behalf of the Cyber Civil Rights Initiative (since its start in
2013). Although that work has largely stopped due to the dismantling described in this piece, Citron
continues to serve on Spotify’s Safety Advisory Council. Introducing the Spotify Safety Advisory Council,
Spotify (June 13, 2022), https://newsroom.spotify.com/2022-06-13/introducing-the-spotify-safety-
advisory-council/.
34 This resulted in several publications focused on privacy law, but also addressing trust and safety as the

isomorphic diffusion of practices from one company to the next. See, e.g., Ari Ezra Waldman, INDUSTRY
UNBOUND (2021); Waldman, False Promise, supra note 27.
35 Jedediah Britton-Purdy, David Singh Grewal, Amy Kapczynski & K. Sabeel Rahman, Building a Law-and-

Political-Economy Framework: Beyond the Twentieth-Century Synthesis, 129 YALE L.J. 1784 (2020).

6
that surveillance-based platforms amassed power in an environment of “lawlessness,” 36
pointing to active legal entrepreneurship by digital platforms to create zones of immunity
around data-extractive business models. 37 Anupam Chander similarly noted that law
helped build Silicon Valley’s power by insulating the information industry from
accountability and allowing companies to collect and monetize data with few limits. 38
Elsewhere, one of us (Waldman) has argued that the practices that have been normalized
as central to privacy law in the United States developed due to uncertainty in laws on the
books, opening opportunities for tech companies themselves to fill in the blanks. 39

Law has also shaped the form and contours of trust and safety. Congress passed
Section 230 of the Communications Decency Act of 1996, which secured a legal shield for
online platforms that left up or removed user-generated content that turned out to be
tortious or otherwise illegal. 40 Top of mind for the law’s drafters, Representatives Chris
Cox and Ron Wyden, was the case of Stratton Oakmont v. Prodigy, 41 a state trial court
decision holding an online service provider strictly liable for trying to remove troubling
content but failing to catch alleged defamation. 42 Representatives Cox and Wyden argued
that Stratton Oakmont created perverse incentives for platforms: If they try to moderate
content and fail to capture everything that should be filtered or removed, they could be
sued for millions, as was the case in Stratton Oakmont; if they did not try, their platforms
would become rife with defamation, abuse, harassment, obscenity, and hate, all of which
could turn off advertisers, speakers, and listeners. 43 Online service providers of the
nascent internet found themselves in a bind.

Federal lawmakers responded with legislation. To avoid the possibility that platforms
would be penalized for trying to “clean up the Internet” but failing to do so perfectly—
either by leaving up or taking down too much third-party speech, 44 Congress passed

36 SHOSHANA ZUBOFF, THE AGE OF SURVEILLANCE CAPITALISM 104 (2019) (“[L]awlessness has been a critical
success factor in the short history of surveillance capitalism.”); id. at 103 (“A key element of Google’s
freedom strategy was its ability to discern, construct, and stake its claim to unprecedented social territories
that were not yet subject to law.”).
37 JULIE E. COHEN, BETWEEN TRUTH AND POWER: THE LEGAL CONSTRUCTIONS OF INFORMATIONAL CAPITALISM

5-6 (2019).
38 Anupam Chander, How Law Made Silicon Valley, 63 EMORY L.J. 639 (2014).
39 WALDMAN, supra note 34, at 120-126; Waldman, False Promise, supra note 27, at 793-798.
40 47 U.S.C. § 230; Danielle Keats Citron, How to Fix Section 230, 103 B.U. L. REV. 714, 723 (2023); see also

CITRON, HATE CRIMES IN CYBERSPACE, supra note 2, at 177-181.

41 No. 31063/94, 1995 WL 323710, at *4-5 (N.Y. Sup. Ct. May 24, 1995).
42 JEFF KOSSEFF, THE TWENTY-SIX WORDS THAT CREATED THE INTERNET 2 (2019); Danielle Keats Citron &

Benjamin Wittes, The Internet Will Not Break: Denying Bad Samaritans § 230 Immunity, 86 FORDHAM L. REV.
401 (2017).
43 Danielle Keats Citron, How to Fix Section 230, 103 B.U. L. REV. 713, 720-23 (2023).
44 Alina Selyukh, Section 230: A Key Legal Shield for Facebook, Google, Is About to Change, NPR (Mar. 21, 2018),

https://www.npr.org/sections/alltechconsidered/2018/03/21/591622450/section-230-a-key-legal-
shield-for-facebook-google-is-about-to-change (quoting former Representative Cox).

7
Section 230 in the Communications Decency Act of 1996.45 Federal courts interpreted
Section 230(c)—the provision key to this discussion 46—as providing a broad immunity,47
with no strings attached. 48

To be clear, Congress expected a quid pro quo: “[I]n return for [not being] sued
indiscriminately,” the companies themselves would be “responsible in terms of policing
platforms.” 49 Congress hoped for “Good Samaritan blocking and filtering offensive
material.” 50 Section 230, Chander argued, created “a legal framework conducive to
promoting speech on the Internet through online speech intermediaries.” 51 Jeff Kosseff
called it “the twenty-six words that created the internet.”52

Whatever its long-term impact, our concerns are more immediate; in the short term,
Section 230 mandated self-governance without any guidance. Under the law, platforms
were left to determine what should and should not be published online. In the process of
answering one question—namely, whether platforms should be treated as publishers of
defamatory content with strict liability, the question at issue in Stratton Oakmont—Section
230 raised many more questions: How should platforms “block[] and filter[]” content?
What constitutes “offensive” material? Who should determine what gets blocked or
filtered? The hands-off approach embodied by Section 230 provided no answers.

This uncertainty had two effects. First, because they were on their own, leading
platforms developed content moderation rules and practices from the ground-up; these
rules were necessarily inflected by the companies’ independent priorities. Second, many
platforms learned from and copied each other, adopting what they perceived as accepted
practices that had been legitimized and normalized over time by the first movers in the
field. Therefore, legal uncertainty helped catalyze both the growth and diffusion of what
would later be called trust and safety. We turn to these effects in the next two sections.

45 Citron, How to Fix Section 230, supra note 40, at 722-23.

46 Section 230(c) has two subparts. Section 230(c)(1), which is entitled “Treatment as Publisher or Speaker,”
provides that online service providers will not be treated as publishers or speakers of information provided
by third parties. 47 U.S.C. 230(c)(1). Section 230(c)(2), which is entitled “Civil Liability,” says that online
service providers will not face civil liability for filtering, blocking, or removing content so long as they do
so in good faith. 47 U.S.C. 230(c)(2).
47 See Zeran v. Am. Online, Inc., 129 F.3d 327 (4th Cir. 1997); Doe v. Internet Brands, Inc., 824 F.3d 846, 851-

52 (9th Cir. 2016); Jane Doe No. 1 v. Backpage.com, LLC, 817 F.3d 12, 18 (1st Cir. 2016) (citing cases from
the First, Fifth, Ninth, and Eleventh Circuits to show “near-universal agreement” that Section 230(c)(1)
should be interpreted broadly). But see Citron & Wittes, supra note 42, at 404-411 (demonstrating the wide
gap between judicial interpretation and the original intent of Congress when enacting Section 230(c)(1)).
48 Citron, How to Fix Section 230, supra note 40, at 723-29; Citron & Wittes, supra note 42, at 408-410.
49 Selyukh, supra note 44 (quoting Senator Wyden).
50 The title of Section 230(c) is “Good Samaritan blocking and filtering offensive material.” 47 U.S.C. § 230(c).
51 Chander, supra note 38, at 657.
52 Kosseff, supra note 42, at 3-4. The twenty-six words to which Kosseff refers is Section 230(c)(1), hence

some of the confusion in general discourse around Section 230, whose provisions (a-f) contain hundreds of
words.

8
B. The Endogenous Development of Trust and Safety Policies

When there is uncertainty about the law’s requirements, some organizations look
inward, creating routines, practices, and procedures to fill in the gaps. 53 This is, after all,
one of the jobs of in-house lawyers and compliance professionals—to filter and translate
the law on the books into corporate structures and behaviors in ways that support
corporate values and shareholder goals. 54 That process can help advance the purpose of
the law or it can frustrate it, as when any legal obligation is filtered through the prime
directive of corporate profits. 55 Sociolegal scholar Lauren Edelman identified this link
between uncertainty in the law on the books and the creative, ground-up work of
corporate employees as part of a larger social phenomenon known as “legal
endogeneity.” 56

The rules and practices of trust and safety were developed by platforms; those rules
and practices evolved in accordance with platforms’ growth, engagement, and
independent priorities. Trust and safety evolved over time. It began with formal rules
and then added structural interventions. Given the social and political pressures faced by
the fastest growing platforms, it is no surprise that those platforms led the way.

Early content moderation was ad hoc and informal. 57 Customer service often handled
people’s concerns. 58 In 2003, when Facebook was only available to people with college
emails, student volunteers tackled user complaints. 59 The bespoke nature of this work
created few incentives for companies to reduce their decisions to routinized and formal

53 See LAUREN EDELMAN, WORKING LAW 30-31 (2016); Waldman, False Promise, supra note 27, at 803-810.
Laws on the books can be vague for many reasons, including the inherent limitations of language and the
legislative drafting process. See, e.g., Victoria F. Nourse & Jane S. Schacter, The Politics of Legislative Drafting:
A Congressional Case Study, 77 N.Y.U. L. REV. 575, 594-96 (2002) (documenting “deliberate ambiguity” in
statutes).
54 EDELMAN, WORKING LAW, supra note 53, at 31.
55 Id. at 32-33. This process is one reason why the law on the books can differ from the law on the ground.

See Jon B. Gould & Scott Barclay, Mind the Gap: The Place of Gap Studies in Sociolegal Scholarship, 8 ANN. REV.
L. & SOC. SCI. 323 (2012).
56 EDELMAN, WORKING LAW, supra note 53, at 12. One of us (Waldman) has argued that a form of legal

endogeneity explains the rise, evolution, and ineffectiveness of proceduralist privacy law. See WALDMAN,
supra note 34; Waldman, False Promise, supra note 27. Legal endogeneity theory includes an important role
for the judiciary, which may be “induced” into or ideologically committed to incorporating endogenously
developed corporate structures as evidence of actual compliance with the law. Edelman, Uggen, &
Erlanger, supra note 27, at 408. We omit this part of Edelman’s institutionalist theory because it is not
relevant to the initial development of trust and safety practices.
57 GILLESPIE, supra note 1, at 17.
58 Jeffrey Rosen, The Delete Squad, THE NEW REPUBLIC (Apr. 29, 2013),
https://newrepublic.com/article/113045/free-speech-internet-silicon-valley-making-rules; Jeffrey Rosen,
Google’s Gatekeepers, N.Y. TIMES MAG. (Nov. 28, 2008),
https://www.nytimes.com/2008/11/30/magazine/30google-t.html.
59 GILLESPIE, supra note 1, at 119.

9
policies. Instead, it was often in-house counsel who determined whether particular
sensitive content should remain online or be removed. In the early 2000s, YouTube
employees had the first pass at figuring out if videos should be taken down or sent for
“specialized review.” 60 Politically sensitive questions made their way to Google’s then-
Vice President and Deputy General Counsel Nicole Wong whose high-profile calls
earned her the moniker “The Decider.” 61 Hemanshu Nigam, a former prosecutor, played
a similar role at MySpace. Hired as the Chief Security Officer in 2005, Nigam made key
decisions about prohibited speech, including hate speech and bullying, so the site could
be “family friendly.” 62

The biggest tech companies responded to exogenous political and social pressures by
turning to their internal teams. 63 Those teams looked to legal norms to set more formal
rules around what users could say and do. 64 Companies then adopted formal speech
policies and trained employees (and later contractors) to respond to reported violations. 65
Facebook hired its first full-time employee devoted to content policy in 2008. When Dave
Willner joined Facebook’s content policy team in 2010 after working nights on the help
desk, the site had no formal speech rules. 66 Willner volunteered to draft them, relying on
university anti-harassment codes and insights from philosopher John Stuart Mill whom
he read in college. 67 His team developed policies on sexual content that resembled vague

60 Rosen, Google’s Gatekeepers, supra note 58.

61 Id.
62 Danielle Keats Citron & Helen Norton, Intermediaries and Hate Speech: Fostering Digital Citizenship for our

Information Age, 91 B.U. L. REV. 1435, 1454 (2011) (discussing telephone interview with Nigam); see also
Hemanshu (Hemu) Nigam, Partner, https://www.venable.com/professionals/n/hemanshu-nigam (last
visited June 24, 2025).
63 CITRON, HATE CRIMES IN CYBERSPACE, supra note 2, at 227-236.
64 DAVID KAYE, SPEECH POLICE: THE GLOBAL STRUGGLE TO GOVERN THE INTERNET 53-58 (2019) (describing

biweekly “mini-legislative sessions” led by Facebook’s head of global policy Monika Bickert with
company’s 50 or so members of the content policy teams to address speech policy questions in 2018).
65 Rosen, Google’s Gatekeepers, supra note 58. See SARAH ROBERTS, BEHIND THE SCREEN: CONTENT MODERATION

IN THE SHADOWS OF SOCIAL MEDIA 73-101 (2021); Adrian Chen, Facebook Releases New Content Guidelines,
Now Allows Bodily Fluids, GAWKER (Feb. 16, 2012),
https://web.archive.org/web/20121010194210/http://gawker.com/5885836/facebook-releases-new-
content-guidelines-now-allows-bodily-fluids; Helen A.S. Popkin, How Facebook Keeps the Porn, Gore and Hate
Out of Your News Feed, NBC NEWS.COM (Feb. 17, 2012),
https://www.nbcnews.com/technology/technolog/how-facebook-keeps-porn-gore-hate-out-your-
news-feed-157748.
66 Klonick, supra note 15, at 1620.
67 Rosen, The Delete Squad, supra note 58. As Dave Willner discussed with one of us (Citron), Mill’s

dedication to free expression limited by the harm principle was front of his mind. Willner’s wife, Charlotte
Willner, worked in Facebook’s user safety group, which addressed child safety and suicide prevention. Id.
Charlotte Willner would later go on to run trust and safety at Pinterest and then to lead the Trust and Safety
Professional Association in 2020, which we discuss later in this Part.

10
anti-vice morality laws. 68 At the time, Facebook’s content policy team had six employees,
who were located in Menlo Park, California; a few hundred employees, working in the
United States, Ireland, and India, “reviewed complaints about nudity, porn, hate speech,
and violence.” 69 To enforce its community guidelines, Facebook hired people with
expertise in different languages and political and cultural understandings. 70

Twitter exemplifies the evolution from rules alone to rules and design, further
demonstrating the endogenous nature of these practices. That evolution proceeded
slowly. In 2008, Twitter hired Del Harvey as its first safety employee; Harvey previously
served as a law enforcement liaison for a nonprofit fighting child sexual exploitation.71
Twitter’s first speech policy, adopted in 2009, said Twitter would not “actively monitor”
or “censor” users’ content unless it involved spam, impersonation, or copyright.72
Harvey, having faced online abuse herself, wanted the company to ban attacks on
individuals so she asked one of us (Citron) to write a memorandum describing the nature
and harms wrought by threats and cyber stalking. 73 Harvey discussed the issues with the
C-Suite, but she was told that the site’s hands-off policy would continue. 74 It was a matter
of priorities, as Harvey reflected in 2023: “I made requests in 2010 for functionalities [to
address abusive practices like violators returning to the platform by creating new
accounts] that did not get implemented, in many instances, ‘til a decade-plus later.” 75 The
reason was the site’s “growth at all costs” mentality, and safety was, to her bosses, too
much downside and too little upside. 76 Safety by design was filtered through the self-
interested motives of the company and, at least at the time, those interests weakened trust
and safety efforts.

68 Waldman, Disorderly Content, supra note 10, at 923-950 (arguing that community guidelines for sexual
content appear and are implemented in ways more akin to morality laws like lewdness and anti-vice
ordinances than liberal free speech rules).
69 Rosen, The Delete Squad, supra note 58.
70 GILLESPIE, supra note 1, at 119.
71 Del Harvey, Delibius.com, https://delbius.com/. Harvey stayed at Twitter for thirteen years, leading

Twitter’s trust and safety team until she retired in 2021. Lauren Goode, Twitter’s Former Head of Trust and
Safety Finally Breaks Her Silence, WIRED (Nov. 21, 2023), https://www.wired.com/story/del-harvey-twitter-
trust-and-safety-breaks-her-silence/. When Harvey joined Twitter in 2008, she was employee number 25.
Id. Harvey “was the first executive put in charge of spam and abuse, which later became ‘trust and safety.’”
Id. Yoel Roth took over as head of trust and safety when Harvey left Twitter in October 2021. Id.
72 The Twitter Rules, TWITTER SUPPORT (Jan. 18, 2009),
https://web.archive.org/web/20090118211301/ttp://twitter.zendesk.com/forums/26257/entries/1831
1; Sarah Jeong, The History of Twitter’s Rules, VICE (Jan 14, 2016), https://www.vice.com/en/article/the-
history-of-twitters-rules/.
73 Danielle Keats Citron & Hany Farid, This is the Worst Time for Donald Trump to Return to Twitter, SLATE

(Nov. 20, 2022), https://slate.com/technology/2022/11/trump-returning-to-twitter-elon-musk.html.

74 Tony P. Love et al., Analyzing Virtual Manhood: Qualitative Analysis of Fappening-Related Twitter Data, Sage,

https://methods.sagepub.com/case/virtual-manhood-qualitative-analysis-of-fappening-related-twitter-
data.
75 Goode, supra note 71.
76 Id.

11
 This development of trust and safety practices occurred in the shadow of Section 230’s
legal uncertainty. The law’s drafters may have hoped to incentivize good corporate
behavior, but what this section has shown—and what legal endogeneity theory would
predict—is that trust and safety practices developed endogenously not necessarily in line
with the public interest but rather in response to corporate self-interest. That is not
inherently bad; corporate and public interests can align. But platforms created trust and
safety practices subject to internal organizational missions and financial and social
pressures rather than based on top-down pressures from the law.

Notably, as this section has suggested, trust and safety did not just grow; it also spread
throughout the ecosystem of digital platforms that host third party content, particularly
social media. That diffusion reflects the second effect of Section 230’s legal uncertainty,
which is the subject of the next section.

C. The Diffusion of Trust and Safety

Where some companies look inward to deal with legal uncertainty, others look
outward: They learn from and copy their competitors. As a result, corporate practices
converge over time, which leads to companies in the same field looking the same. 77 Often,
these practices spread not because they make business sense for everyone; rather, the
practices become a defining social feature of what it means to be part of that field (like
open-plan offices at large law firms). Faced with uncertainty as to what should and
should not be posted online, many content platforms—especially smaller ones—looked
to industry leaders and either learned from those practices or felt compelled to adopt
them.

Organizational sociologists have identified three primary pathways through which

this isomorphic diffusion occurs: coercive, mimetic, and normative. Coercive
isomorphism occurs when organizations respond similarly to similar exogenous forces,
like a new law, common cultural practices, or other outside pressures. 78 Mimetic
isomorphism occurs when organizations adopt the policies and practices of others in their
field because those practices are perceived as legitimate or successful, reducing
problematic legal uncertainty.79 Normative isomorphism refers to situations where
organizations in the same field respond similarly to shared professional norms, 80 often as
a result of professional groups articulating and inculcating those norms as defining

77 DiMaggio & Powell, supra note 28, at 149.

78 Id. at 152.
79 Id. at 153. See, e.g., Charles Tyler & Heather Gerken, The Myth of the Laboratories of Democracy, 122 COLUM.

L. REV. 2187, 2219-2220 (2022); Ronen Shamir, Socially Responsible Private Regulation: World-Culture or World-
Capitalism?, 45 LAW & SOC’Y REV. 313, 317 (2011).
80 DiMaggio & Powell, supra note 28, at 154. See, e.g., Emily Chertoff, Violence in the Administrative State, 112

CALIF. L. REV. 1941, 1948 (2024) (suggesting that workforce pressure, particularly in agencies where front-
line workers are permitted to engage in violence, can drive change in practices).

12
features of what it means to be a professional in a particular field. These three types do
not always have clear boundaries; the same realities may contribute to more than one
means of regression toward the mean.

As popular culture has highlighted and satirized, the tech industry’s copycat behavior
includes open office layouts with bean bags and couches, free meals and snacks, on-site
massages, and game play; 81 these features became synonymous with the social construct
of a “tech company”. 82 Institutional isomorphism is not all frivolity like the free food,
bikes, and volleyball courts dotting Google campuses and other tech companies. 83 We see
thoughtful discussion of the three mechanisms in the legal literature, including
administrative law, 84 international and comparative law, 85 the legal profession, 86 criminal
justice, 87 and privacy. 88

Each form of isomorphic diffusion is evident in the first stages of trust and safety
development. Europe raised the specter of new laws, which exerted pressure on
platforms to expand trust and safety operations. After the terrorist attacks in Paris and
Brussels in 2015, French and Belgian authorities demanded that social media companies
collaborate to make it more difficult for terrorist actors to use their platforms. 89 If they
did not, national and EU legislation would force their hands. 90 Platforms reacted by
creating the Global Internet Forum to Counter Terrorism (GIFCT), a collaborative,
industry-led effort to disrupt terrorist use of social media. 91 Countries paired threats of

81 Superb (and wonderfully funny) examples include HBO television series Silicon Valley and the film The

Internship.
82 As tech journalist Kara Swisher explained, “[o]nce companies like Yahoo and Netscape instituted these

silly environments, everyone copied them, and their indulgent VC parents went along for the ride.” KARA
SWISHER, BURN BOOK 58 (2024). “[M]ost of the headquarters of various internet companies have bouncy
balls, scooters of all kinds, skateboards, and always places to nap.” Id.
83 Yes, one of us (Citron) had the chance to see those perks up close, including at Google, Facebook,

Microsoft, and Twitter.

84 E.g., Donald Langevoort, The SEC as a Lawmaker: Choices About Investor Protection in the Face of Uncertainty,

84 WASH. U. L. REV. 1591, 1610 & n.63 (2006); Jodi Short, The Political Turn in American Administrative Law:
Power, Rationality, and Reasons, 61 DUKE L.J. 1811, 1864 (2012).
85 E.g., Jud Mathews & Alec Stone Sweet, All Things in Proportion? American Rights Review and the Problem of

Balancing, 60 EMORY L.J. 797, 808 (2011).

86 E.g., Laura Beth Nielsen & Catherine R. Albiston, The Organization of Public Interest Practice: 1975-2004, 84

N.C. L. REV. 1591 (2006); Steven Boutcher, Private Law Firms in the Public Interest: The Organizational and
Institutional Determinants of Pro Bono Participation, 1994-2005, 42 LAW & SOC. INQUIRY 543, 561 (2017).
87 E.g., Christina Koningisor, Secrecy Creep, 169 U. PA. L. REV. 1751, 1791-1795 (2021).
88 E.g., Kenneth Bamberger & Deirdre Mulligan, Privacy Decisionmaking in Administrative Agencies, 75 U.

CHI. L. REV. 75, 101 n.108 (2008); Andrew Selbst, An Institutional View of Algorithmic Impact Assessments, 35
HARV. J. L. & TECH. 117, 171-172 (2021).
89 Danielle Keats Citron, Extremist Speech, Compelled Conformity, and Censorship Creep, 93 NOTRE DAME L.

REV. 1035, 1037-1038 (2018).

90 Evelyn Douek, The Rise of Content Cartels, Knight First Amend. Inst., at 9 (2022).
91 Id. at 9-10.

13
new laws with pressure to remove certain posts or accounts; 92 platforms often acceded to
their demands. 93

Although less draconian, platforms also faced pressures from advertisers who argued
that abuse was antithetical to their businesses. Consider Facebook’s evolving hate speech
policy. In 2011, activists protested Facebook’s refusal to view pro-rape pages as hate
speech; the company’s position was that misogynistic speech amounted to humor.94
Advocates Soraya Chemaly and Jaclyn Friedman led a campaign to show advertisers the
harm of misogynistic hate speech, including arguing in op eds that such speech silences
women’s expression and inspires violence. 95 A dozen companies including Nissan
threatened to pull their ads because they did not want their businesses associated with
the glorification of violence against women. 96 Shortly thereafter, Facebook changed its
policies around misogynistic hate speech. 97 Other platforms followed suit: In part
because trust and safety rules and practices were developing endogenously and filtered
through the self-interests of industry, changes came when those interests were affected. 98

Outside pressures similarly led Twitter to change its trust and safety policies in 2014.
Cyber mobs were using Twitter to attack women in gaming (including gaming designers
and CEOs) with doxing, rape and death threats, and defamation, a vicious campaign
known on the platform as #Gamergate. 99 Separately, hacked nude photos of female
celebrities like Jennifer Lawrence and Alyssa Milano began circulating online including
on the platform—the invasions of intimate privacy were dubbed “The Fappening.” 100

92 Elena Chachko, National Security by Platform, 25 STAN. TECH. L. REV. 55, 64 (2021).
93 Vishwam Sankaran, Why did Facebook and Twitter Remove Posts Critical of Indian Government Over Covid
Crisis?, THE INDEPENDENT (Apr. 27, 2021), https://www.the-independent.com/tech/facebook-twitter-
indian-government-posts-b1837643.html.
94 Lizzie Davies, Facebook refuses to take down rape joke pages, GUARDIAN (Sept. 30, 2011),

https://www.theguardian.com/technology/2011/sep/30/facebook-refuses-pull-rape-jokepages
95 Soraya Chemaly, Facebook Rejects Rape Culture. Can You?, CNN (May 31, 2013),
https://www.cnn.com/2013/05/30/opinion/chemaly-facebook. Before adopting the change, Facebook
held an on-site meeting that brought together activists Chemaly and Friedman, academics including one
of us (Citron), and free speech experts to discuss the harms of misogynistic hate speech and the costs to
free expression in changing the site’s policy.
96 CITRON, HATE CRIMES IN CYBERSPACE, supra note 2, at 229.
97 Jessica Valenti, Could the Facebook Win Be Feminism’s Tipping Point, NATION (May 30, 2013),

https://www.thenation.com/article/archive/could-facebook-win-be-feminisms-tipping-point/.
98 GILLESPIE, supra note 1, at 67 (noting that “traffic lights don’t get put in until someone gets hurt at the

intersection”).
99 See, e.g., Nick Wingfield, Feminist Critics of Video Games Facing Threats in ‘GamerGate’ Campaign, N.Y. TIMES

(Oct. 15, 2014), https://www.nytimes.com/2014/10/16/technology/gamergate-women-video-game-

threats-anita-sarkeesian.html; Fruzsina Eordogh, Gamergate and the New Horde of Digital Saboteurs, CHRIS.
SCI. MON. (Jan. 20, 2015), https://www.csmonitor.com/Technology/Tech-
Culture/2014/1125/Gamergate-and-the-new-horde-of-digital-saboteurs.
100 Aly Weisman, Here’s How the Hacked Celebrities Are Responding to Their Nude Phot Leaks, BUS. INSIDER (Sept.

2, 2014), https://www.businessinsider.com/hacked-celebrity-responses-to-nude-photo-leaks-2014-9.

14
 Twitter responded to public outcry about these developments with changes to their
speech policies and enforcement practices. As then-General Counsel (later Chief of Legal,
Policy, and Trust) Vijaya Gadde explained in April 2015: “[C]ertain types of abuse on our
platform have gone unchecked because our policies and product have not appropriately
recognized the scope and extent of harm inflicted by abusive behaviors.”101 Gadde
explained that Twitter was “overhauling [its] safety policies to give … teams a better
framework from which to protect vulnerable users, such as banning the posting of non-
consensual intimate images” and “expanding [the] definition of ‘abuse’ to ban indirect
threats of violence.” 102 Twitter said that it would “triple the size of the team whose job is
to protect users,” which would allow it to “respond to five times the user complaints
while reducing the turnaround time to a fraction of what it was not long ago.”103 The
site’s speech policies “had to change” because no one inside the company “anticipate[d]
all the types of behaviors that [they] would see” on the platform. 104 Gadde explicitly
connected external pressure to internal changes: “We’ve had to move very much from
what we were, which was a platform that very much enabled as much free speech as
possible, to one that is very cognizant of the impact that it’s having on the world, and our
responsibility and our role in shaping that.” 105

Systematic efforts on the design front, however, took far longer to adopt than the
change in speech policies after #Gamergate and The Fappening. 106 As Harvey explained,
Twitter failed to give engineers the time and resources that they needed to build
meaningful safety tools. 107 For instance, Harvey noted, it was years before Twitter got a

101 Vijaya Gadde, Twitter Executive: Here’s How We’re Trying to Stop Abuse While Preserving Free Speech, WASH.
POST (Apr. 16, 2015), https://www.washingtonpost.com/posteverything/wp/2015/04/16/twitter-
executive-heres-how-were-trying-to-stop-abuse-while-preserving-free-speech/.
102 Id. See also Policy and Product Updates Aimed at Combating Abuse, TWITTER OFFICIAL BLOG (2015),

https://blog.twitter.com/official/en_us/a/2015/policy-and-product-updates-aimed-at-combating-
abuse.html (broadening the prohibition from “direct, specific threats of violence against others” to
encompass “threats of violence against others or promot[ing] violence against others”).
103 Gadde, supra note 101.
104 Eric Johnson, Twitter’s Kayvon Beykpour and Vijaya Gadde: The Code Conference interview (transcript), VOX

(June 27, 2019), https://www.vox.com/recode/2019/6/27/18760444/twitter-kayvon-beykpour-vijaya-

gadde-kara-swisher-peter-kafka-code-conference-interview-transcript. No question, enforcement of the
new policies was not perfect. In 2016, cyber mobs targeted actress Leslie Jones with vicious abuse including
nonconsensual intimate images and doctored racist imagery. Katie Rogers, Leslie Jones Becomes a Target of
Online Trolls, N.Y. TIMES (July 19, 2016), https://www.nytimes.com/2016/07/20/movies/leslie-jones-star-
of-ghostbusters-becomes-a-target-of-online-trolls.html. Jones reported the harassment to Twitter but “the
onslaught continued” and Jones quit Twitter. Jack Dorsey responded to Jones directly and admitted that
the site needed to keep continuing its “tools and enforcement systems to prevent this kind of abuse.” Kate
Conger, Harassment of Ghostbusters’ Leslie Jones Shows Twitter Needs to Change, TECHCRUNCH (July 19, 2016),
https://techcrunch.com/2016/07/19/leslie-jones-twitter-harassment/.
105 Johnson, supra note 104.
106 When Jack Dorsey returned as CEO in 2015, he urged Gadde and Kayvon Beykpour, the head of product,

to embed the goal of protecting healthy conversations into the site’s design. Johnson, supra note.
107 Goode, supra note 71.

15
“semi-effective multi-account detection algorithm in place.” 108 Eventually, Twitter was
“one of the first” to assemble a product trust team, at least according to Yoel Roth. 109 That
team designed a feature that gave “people who start a tweet, the ability to moderate
replies within their conversation thread.” 110 Another feature ensured that abusive speech
would not be algorithmically amplified in conversations and notifications. 111 Twitter’s
“Safety Mode,” launched in 2021, blocked or limited violent and harassing replies for
seven days. 112 “[W]hen the feature is turned on in your Settings, our systems will assess
the likelihood of negative engagement” by assessing the Tweet’s content, the
“relationship between the Tweet author and replier,” and “repetitive and uninvited
replies or mentions.” 113

Having dedicated product safety teams spread to Twitter’s competitors in the field.114
By 2019, more and more social media companies embraced the concept of “safety by
design”—building products that are more “resilient to abuse” and that “encourage
civility[,] respect[,] kindness and authenticity.” 115

Mimetic isomorphism is probably the most important diffusion pathway for trust and
safety, especially given the uncertainty created by Section 230. Established platforms’
speech policies often served as templates for new entrants. 116 Because startups lacked
personnel or expertise in content moderation, they followed the lead of established
sites. 117 “[S]maller sites look[ed] to larger ones for guidance, sometimes borrowing
language and policies wholesale” on the assumption that those policies were “battle-

108 Id.
109 Safe Space: Talking Trust and Safety with Yoel Roth, THORN BLOG (Aug. 20, 2024),
https://safer.io/resources/safe-space-talking-trust-safety-with-yoel-roth/.
110 Johnson, supra note 104.
111 Id.
112 Jarrod Doherty, Introducing Safety Mode, TWITTER BLOG (Sept. 1, 2021),
https://blog.x.com/en_us/topics/product/2021/introducing-safety-mode.
113 Id.; What’s ‘Safety Mode’ & How Does it Work? Inside Twitter’s New Feature Aimed to Combat Online

Harassment, ECON. TIMES (INDIA) (Sept. 2, 2021),

https://economictimes.indiatimes.com/magazines/panache/whats-safety-mode-how-long-will-the-
user-be-blocked-inside-twitters-new-feature-aimed-to-combat-online-
harassment/articleshow/85864293.cms
114 At Facebook and elsewhere, those teams worked to detect and remove coordinated inauthentic activity,

which could be spammers or foreign malign influence campaigns. In response to Russian interference with
the elections in 2016 and 2018, Facebook’s head of cybersecurity policy, Nathaniel Gleicher, worked with
investigators and engineers to detect and stop “coordinated inauthentic behavior.” The Facebook Dilemma:
Interview with Nathaniel Gleicher, PBS FRONTLINE,
https://www.pbs.org/wgbh/frontline/interview/nathaniel-gleicher/. Twitter similarly worked to find
and remove fraudulent and inauthentic accounts. Thor Benson, Yoel Roth on Elon Musk Gutting Twitter’s
Election Integrity Team and What it Means for 2024, PUBLIC NOTICE (Oct. 6, 2023),
https://www.publicnotice.co/p/yoel-roth-interview-twitter-elon-musk-elections.
115 Safe Space, supra note 109.
116 GILLESPIE, supra note 1, at 71-72.
117 Id.

16
tested.” 118 For instance, Instagram adopted Flickr’s community guidelines. 119 The trust
and safety community puts together conferences where one of the explicit goals is to give
all participants “a picture of how other types of companies approach the challenges you
face.” 120

This is not to suggest that new platforms copied from a single model. Several market
leaders took different approaches, so some startups copied ones with few restrictions
while others adopted more restrictive ones.121 Social media companies interpreted their
speech policies and enforced them in their own ways. 122 The underlying rules, policies,
and practices spread from established players to new entrants; newcomers trusted
established firms to know what they were doing.

Collaborations among tech companies and outside experts also helped spread trust
and safety practices. In 2016, Twitter created a Trust and Safety Council, which provided
feedback to the policy and product teams on calls and at in-person meetings. 123 The
Council was made up of civil rights, human rights, civil liberties, and child exploitation
experts, including the Cyber Civil Rights Initiative.124 Meta established a Safety Advisory
Council made up of eight internet safety organizations from Brazil, India, Australia, and

118 Id. at 72.

119 Id. at 71.
120 Trust & Safety Curriculum, TSPA, https://www.tspa.org/curriculum/ts-curriculum/ (last visited Aug.

8, 2025).
121 Gillespie calls this mimicry “institutional plagiarism.” GILLESPIE, supra note 1, at 72. Similarities continue

to this day. A 2022 study of content policies at the 43 largest online platforms found that 83% referred to
“community, trust, and safety” as their motivation for banning hate speech, targeted harassment, and
misleading content; more than 95% of those platforms used illustrative examples to explain their policies;
most informed users that they waived their right to complain about speech policies. Brennan Schaffner et
al., “Community Guidelines Make this the Best Party on the Internet”: An In-Depth Study of Online Platforms’
Content Moderation Policies, ACM (May 11-16, 2024), https://doi.org/10.1145/3613904.3642333.
122 One of us (Citron) worked closely with Twitter, Facebook, TikTok, Twitch, and Bumble on internal

guidelines until, as discussed later in this part, her advice was no longer needed. She remains on Spotify’s
Safety Advisory Council.
123 Patricia Cartes, Announcing the Twitter Trust & Safety Council, TWITTER BLOG (Feb. 9, 2016),

https://blog.x.com/en_us/a/2016/announcing-the-twitter-trust-safety-council; Louise Matsakis, Twitter

Trust and Safety Advisers Say They’re Being Ignored, WIRED (Aug. 23, 2019),
https://www.wired.com/story/twitter-trust-and-safety-council-letter/ (discussing the council’s
relatively fruitful relationship with Twitter for first two years of its existence “with regular calls with
company executives to discuss new policies well in advance of their rollouts.”).
124 Cat Zakrzewki, Joseph Menn & Naomi Nix, Twitter Dissolves Trust and Safety Council, WASH. POST (Dec.

12, 2022), https://www.washingtonpost.com/technology/2022/12/12/musk-twitter-harass-yoel-roth/.

CCRI President Dr. Mary Anne Franks and CCRI Vice President Citron served on the Twitter Trust and
Safety Council from 2016 until its dissolution on December 12, 2022. In the first two years of the Council’s
existence, Twitter held regular calls and hosted annual in-person summits. Matsakis, supra note 123. Yet
things deteriorated over time. In 2019, a group of Council members wrote to Twitter to express concern
that months had gone by without updates or consultation about policy and product changes. Id. After the
Council pushed back on this approach, Twitter changed course and began consulting the group earlier in
the product design process. CITRON, FIGHT FOR PRIVACY, supra note 2, 171-72; Doherty, supra note 112.

17
other countries. 125 In 2020, TikTok launched its Content Advisory Council whose
founding members included computer scientist and CCRI Board member Hany Farid
and legal scholar and CCRI President Mary Anne Franks. 126 TikTok later expanded that
group to include members across the world and established the first global Youth
Council. 127 Other collaborations were more retail than wholesale. For instance, in 2018,
Meta’s Zuckerberg “invited more than a dozen professors and academics to a series of
dinners at his home to discuss how Facebook could better keep its platform safe from
election disinformation, violent content, child sexual abuse material, and hate speech.” 128
These collaborations allowed platforms to learn from academics, for academics to learn
from platforms, and for platforms to learn from each other by consulting with the same
academics, further spreading different approaches to online safety.

Some internal collaborations with experts expanded to include other companies. In

2017, Facebook formed the Nonconsensual Intimate Imagery Task Force, led by Global
Head of Safety Antigone Davis and then-Global Safety Policy Director Karuna Nain.
Holly Jacobs, CCRI’s founder, Franks, CCRI’s President, and one of us (Citron) served on
the task force along with Australia’s e-Safety Commissioner Julie Grant and
representatives from the National Network to End Domestic Violence. 129 The Task Force
provided advice to Facebook as it built a hash database of nonconsensual intimate images
that would prevent their re-uploading (or re-appearance) on Facebook and Instagram.130
Later, in partnership with StopNCII.org, Facebook invited other companies to participate
in the hash database. 131 Facebook, Instagram, Threads, TikTok, Reddit, Snap, OnlyFans,
Microsoft Bing, Bumble, Pornhub, bitly, Redgifs, Patreon, and vivastreet joined the
venture.132 Tech companies worked on other initiatives to combat extremist and terrorist

125 Learn More About the Meta Safety Advisory Council, Meta,
https://www.facebook.com/help/222332597793306/ (last visited Aug. 5, 2025). The Safety Advisory
Council released a public letter criticizing Meta for its dismantling of safety efforts. Will McCurdy, Meta’s
Safety Advisory Council Says It’s Prioritizing Politics Over Safety, PC MAG. (Feb. 1, 2025),
https://www.pcmag.com/news/metas-safety-advisory-council-says-company-prioritizing-political-
ideologies.
126 Introducing the TikTok Content Advisory Council, Mar. 18, 2020, https://newsroom.tiktok.com/en-

us/introducing-the-tiktok-content-advisory-council.
127 Engaging Our Advisory Councils, TikTok, https://www.tiktok.com/transparency/en-us/advisory-

councils/ (last visited Aug. 5, 2025) (noting that the U.S. Advisory Council includes Mutale Nkonde, Dawn
Nunziato, Desmond Upton Patton, and several others in addition to Farid and Franks).
128 Jason Koebler, Has Facebook Stopped Trying?, 404 MEDIA (June 24, 2024), https://www.404media.co/has-

facebook-stopped-trying/. Citron was at one of those dinners.

129 CITRON, FIGHT FOR PRIVACY, supra note 2, at 176-78; Antigone Davis, Detecting Non-Consensual Intimate

Images and Supporting Victims, Meta (Mar. 15, 2019), https://about.fb.com/news/2019/03/detecting-non-

consensual-intimate-images/.
130 Citron, Sexual Privacy, supra note 2, at 1956-57.
131 About, Stop NCII.org, https://stopncii.org/faq/?lang=en-gb (last visited Aug. 8, 2025).
132 Industry Partners, StopNCII.org, https://stopncii.org/partners/industry-partners/?lang=en-gb (last

visited Aug. 8, 2025).

18
content, suicide, and misinformation generated by AI.133 These collaborations across
companies brought many trust and safety employees in contact with each other, further
building and sharing knowledge.

There has also been an industry-led effort to articulate best practices. In February
2021, the Digital Trust and Safety Partnership was formed by Discord, Google, LinkedIn,
Meta, Microsoft, Pinterest, Reddit, Twitter, and Vimeo. 134 DTSP described itself as a
“first-of-its-kind industry driven initiative” to articulate and promote best practices for
trust and safety. 135 The group issued a trust and safety glossary of terms 136 and outlined
best practices for the use of artificial intelligence and automation. 137 Of course, these best
practices are imperfect; we might have drafted them differently. Our point is narrower—
namely, that companies came together and adopted many similar policies as they looked
inward and outward for direction on how to build safer digital environments.

Tech companies also voluntarily partnered with governments to tackle destructive

online activity. After a terrorist attacked two mosques in Christchurch, New Zealand and
livestreamed himself murdering 51 people and injuring 50 others, New Zealand Prime
Minister Jacinda Ardern and French President Emmanuel Macron brought together
heads of state and tech company leaders. 138 That meeting led to the creation of the
nonprofit Christchurch Call. 139 Amazon, Facebook, Google, Dailymotion, Microsoft,
Qwant, Twitter, and YouTube joined the group, which relaunched GIFCT to share
knowledge on terrorists’ use of platforms. 140 GIFCT assisted smaller online platforms in
countering online terrorist activity.141 Christchurch Call grew to include nineteen online

133 Goode, supra note 71. Twitter’s Del Harvey explained that collaborations across companies usually
involved issues where there was considerable consensus, starting with child sexual exploitation and then
moving to terrorism and suicide and then to misinformation involving synthetic content. Id.
134 Joseph Menn et al., Chinese Propaganda Accounts Found by Meta Still Flourish on X, WASH. POST (Feb. 16,

2024), https://www.washingtonpost.com/technology/2024/02/16/x-meta-china-disinformation/.
135 David Sullivan, A Portfolio Approach to Trusted Intermediaries for Online Content and Conduct, COL. TECH.

POLICY REV. (2025).

136 Press Release, The Digital Trust & Safety Partnership Releases Inaugural Trust & Safety Glossary of

Terms (July 13, 2023), https://dtspartnership.org/press-releases/the-digital-trust-safety-partnership-

releases-inaugural-trust-and-safety-glossary-of-terms/.
137 Press Release, DTSP Unveils New Best Practices for Incorporating AI and Automation Into Trust and

Safety (Sept. 24, 2024), https://dtspartnership.org/press-releases/dtsp-unveils-new-best-practices-for-

incorporating-ai-and-automation-into-trust-and-safety/.
138 Significant Progress Made on Eliminating Terrorist Content Online, The Christchurch Call, Sept. 24,

2019, https://www.christchurchcall.org/significant-progress-made-on-eliminating-terrorist-content-
online/.
139 Id.
140 Id.
141 Id. In the mid 2010s, the U.S. government regularly shared counterintelligence threat information with

tech companies; indeed, Meta’s CEO Mark Zuckerberg specifically requested government information
sharing after Russia’s Internet Research Agency used its platform to spread disinformation in the run up
to the 2016 U.S. election. Danielle Keats Citron & Jeff Stautberg, Public-Private Partnerships After Murthy v.
Missouri, 100 INDIANA L.J. 791, 810 (2025). After the 2016 election, federal agencies shared intelligence with

19
service providers, 13 partner organizations, and 50 civil society organizations. 142 Platform
practices changed in similar ways as a result of this coordination.

Normative isomorphism likely also played a role but is harder to identify the precise
pathways through which trust and safety norms spread. We know that when Twitter and
Facebook announced that their speech policies and practices would be paired with
structural, design-oriented efforts, other tech companies began to treat destructive online
activity “as a holistic problem” requiring coordination of policy, legal, and product
teams. 143 That development likely grew out of individuals sharing and copying other
companies’ practices. But it also necessarily involved changes inside companies. Trust
and safety teams grew to include engineers, policy experts, social scientists, and former
law enforcement agents, 144 each of whom brought different professional norms to their
work. Yoel Roth, formerly Twitter’s Head of Safety and Integrity, explained that in his
fifteen years in trust and safety, the field expanded beyond content moderation, the so-
called “janitors of the internet,” to include designing safer products and trading ideas
with each other. 145

Trust and safety grew into a profession that articulated norms of holistic safety and
collaboration at conferences. By the mid-2010s, there were hundreds of trust and safety
employees in Silicon Valley who formed a “tight social and professional circle.” 146 In
2018, academics and industry leaders organized the first trust and safety conference,
Content Moderation at Scale, at Santa Clara University; there were over 200 attendees

tech companies about misinformation concerning the “mechanics of voting that were circulating online.”
Id. at 811. Public-private coordination stalled in the wake of litigation brought by the Missouri Attorney
General against federal agencies for allegedly coercing tech companies to take down disfavored content
such as COVID disinformation; lower court rulings enjoined those partnerships on the grounds of
jawboning in violation of the First Amendment, but those rulings were ultimately vacated by the U.S.
Supreme Court. Id. at 804. It is unclear the extent to which federal agencies have been sharing information
with social media companies during the second Trump administration; Palantir’s partnership with the
Trump Administration raises concerns about privacy and civil liberties that we leave to discuss in other
work. Bobby Allyn, How Palantir, the Secretive Tech Company, is Rising in the Trump Era, NPR (May 3, 2025),
https://www.npr.org/2025/05/01/nx-s1-5372776/palantir-tech-contracts-trump.
142 Announcing the Foundation, The Christchurch Call, May 14, 2024,
https://www.christchurchcall.org/christchurch-call-foundation-announcement/.
143 Id.
144 Christine Lehane, A Career in Trust and Safety: You know more than you know, MEDIUM (June 4, 2022),

https://medium.com/@christinemlehane/a-career-in-trust-safety-you-know-more-than-you-know-
3a02f63059a4; Christine Lehane, UX Researcher/Product Researcher/ Design Researcher/ Market Researcher/
Research Scientist…Which one am I?, MEDIUM (June 21, 2022), https://medium.com/@christinemlehane/ux-
researcher-product-researcher-design-researcher-market-researcher-research-scientist-2e0a5757a549;
Christine Lehane, Research in the Context of Highly Sensitive Topics/Logistical Challenges, MEDIUM (Sept. 2,
2022), https://medium.com/@christinemlehane/research-in-the-context-of-highly-sensitive-topics-
logistical-challenges-b1e733def84b. See also Trust & Safety, All Tech is Human,
https://alltechishuman.org/trust-safety (last visited Aug. 17, 2025).
145 Safe Space, supra note 109.
146 GILLESPIE, supra note 1, at 71-72, 118.

20
from multiple companies, including Match, Twitter, Twitch, Google, and Facebook.147
That conference led to three others in Washington, D.C. (2018), New York City (2018),
and Brussels (2019). 148 In September 2022, TSPA held the inaugural TrustCon conference,
which was held in Palo Alto, California. 149 The conference had 800 participants from over
200 companies across more than 40 countries. 150 It welcomed industry practitioners as
well as individuals from government, academia, and elsewhere. 151 By the end of 2022,
TSPA had 2,000 active members. 152 At these conferences, participants did not simply talk
about content moderation; topics included transparency, design, employee hiring, and
vendors.153 Companies gave overviews of their own operations in full view of their
competitors, both expressing the collaborative norms of the profession and hoping to
learn from one another. 154

The spread of holistic norms around trust and safety also tracked the establishment
of the field’s professional organization in 2020. That year, Eric Goldman, Clara Tsao, and
Adelin Cai formed the Trust and Safety Professional Association (TSPA) and the Trust &
Safety Foundation (TSF). TSPA’s Board included Alex Macgillivray, Twitter’s former
general counsel, and Danelle Dixon, former COO of the Mozilla Foundation. 155 Charlotte
Willner, formerly of Facebook and Pinterest, stepped into the role of founding Director
of the TSPA in November 2020. 156 As she explained: “Trust and safety is still a relatively
new profession, and practitioners historically have learned by doing, with few
opportunities for formal training. TSPA’s mission is to support our peers in developing
the expertise and skills they need to navigate this high-stakes environment.” 157 TSPA was

147 Eric Goldman, COMO: Content Moderation at Scale Conference Recap, TECH. & MARKETING LAW BLOG (July
16, 2018), https://blog.ericgoldman.org/archives/2018/07/como-content-moderation-at-scale-
conference-recap.htm.
148 Eric Goldman, A Pre-History of the Trust & Safety Professional Association, TECH. & MARKETING LAW BLOG

(June 17, 2020), https://blog.ericgoldman.org/archives/2020/06/a-pre-history-of-the-trust-safety-

professional-association-tspa.htm.
149 Trust & Safety Professional Association Announces TrustCon, the First Global Conference for Trust and

Safety Professionals, September 26, 2022, TSPA, https://www.tspa.org/2022/09/26/trust-safety-

professional-association-announces-trustcon/ (last visited Aug. 8, 2025).
150 Id.
151 Id.
152 Citizens and Tech, “Two Years at the Trust and Safety Professionals Association-TrustCon 2022”,

https://citizensandtech.org/2022/09/tspa-two-years-on/ (last visited Aug. 5, 2025).

153 Id.
154 See, e.g., Eric Goldman, Roundup of February’s ‘COMO at Scale Brussels’ Event, TECH. & MARKETING LAW

BLOG (Apr. 29, 2019), https://blog.ericgoldman.org/archives/2019/04/roundup-of-februarys-como-at-

scale-brussels-event.htm. A video recording of these presentations is available at
https://santaclarauniversity.hosted.panopto.com/Panopto/Pages/Viewer.
155 Goldman, Pre-History, supra note 148.
156 Announcing Charlotte Willner as Founding Executive Director of TSPA and TSF, TSPA (Nov. 19, 2020),

https://www.tspa.org/2020/11/19/announcing-charlotte-willner-as-founding-executive-director-of-
tspa-and-tsf/.
157 Charlotte Willner, Introducing the Trust and Safety Curriculum, TSPA BLOG (June 17, 2021),

https://www.tspa.org/2021/06/17/introducing-the-trust-and-safety-curriculum/.

21
set up as a “nonprofit, membership-based organization” to support the “global
community of professionals who develop and enforce principles and policies that define
acceptable behavior online.” 158 It would serve as a “forum for professionals to connect
with a network of peers, find resources for career development, and exchange best
practices;” it would “develop guides on the strengths and weaknesses of technical
tools.” 159 TSPA’s founding corporate supporters were Airbnb, Automattic, Cloudflare,
Facebook, Instagram, WhatsApp, Google, YouTube, Match Group, Omidyar Network,
Pinterest, Postmates, Slack, Twitter, and Wikimedia Foundation. 160 Collaboration was
clearly at the heart of the TSPA.

The trust and safety field also expresses its professional norms through the pages of
the Journal of Online Trust and Safety, which is published by the Stanford Internet
Observatory. 161 The journal is peer-reviewed and inter-disciplinary; the editorial board
includes scholars from media studies, computer science and law. 162 From the very
beginning and continuing through its volumes today, JOTS has published scholarly
articles on a broad range of topics reflecting the organization’s holistic view of trust and
safety. Articles have explored hashing technology, 163 transparency reporting,164
misinformation and elections, 165 online harassment, 166 and the profession itself, 167 among
many other parts to the trust and safety puzzle.

Through collaborations, overt mimicry, and shared norms, trust and safety practices
spread throughout the ecosystem of tech platforms. Those practices have evolved from
ad hoc decisions to comprehensive rules to holistic approaches to designing healthy
digital environments. But these developments are now in the rear-view mirror. As the
next Part shows, trust and safety is entering a new phase in large part because of law, and
those changes are not necessarily for the better.

158 Adelin Cai, Alexander Macgillivray, Clara Tsao, Denelle Dixon, & Eric Goldman, New Organization
Dedicated to Online Trust and Safety, TSPA (June 17, 2020), https://www.tspa.org/2020/06/17/new-
organizations-dedicated-to-online-trust-and-safety/.
159 Id.
160 Id.
161 See Announcing the Journal of Online Trust and Safety, STANFORD FREEMAN SPOGLI INSTITUTE (July 29,

2021), https://reap.fsi.stanford.edu/news/announcing-journal-online-trust-and-safety.
162 Id.
163 Hany Farid, An Overview of Perceptual Hashing, 1 J. ONLINE TRUST & SAFETY 24 (2021).
164 Camille Francois & evelyn douek, The Accidental Origins, Underappreciated Limits, and Enduring Promises

of Platform Transparency Reporting about Information Operations, 1 J. ONLINE TRUST & SAFETY 17 (2021).
165 John Ternovski, Joshua Kalla, & P.M. Aronow, The Negative Consequences of Informing Voters About

Deepfakes: Evidence from Two Survey Experiments, 1 J. ONLINE TRUST & SAFETY 28 (2022).
166 Ifat Gazia, Trevor Hubbard, Timothy Scalona, Yena Kang, & Ethan Zuckerman, Proactive Blocking through

the Automated Identification of Likely Harassers, 2 J. ONLINE TRUST & SAFETY 175 (2024); Timothy J. Foley &
Melda Gurakar, Backlash or Bullying? Online Harassment, Social Sanction, and the Challenge of COVID-19
Misinformation, 1 J. ONLINE TRUST & SAFETY 31 (2021).
167 Riana Pfefferkorn, Content-Oblivious Trust and Safety Techniques: Results from a Survey of Online Service

Providers, 1 J. ONLINE TRUST & SAFETY 14 (2022).

22
 II. TRUST AND SAFETY’S EVOLUTION

This Part challenges the conventional wisdom about the current state of trust and
safety. It is true that tech companies in the United States are firing employees, changing
policies, and pulling back on safety promises. These developments have triggered
narratives of collapse based on contracting budgets. The reality is more complex. As we
show, trust and safety endeavors are shifting from an amalgam of decisions, rules, and
design strategies triggered by legal uncertainty to a public-private partnership in which
procedural compliance constitutes tech companies’ primary responsibilities. This
evolution is due to legal developments in the European Union and the natural pull of
compliance-as-governance throughout the tech ecosystem. Trust-and-safety-as-
compliance is already spreading, but the full effects—on jobs, on digital environments,
on the social media ecosystem—remain to be seen.

A. The Narrative of Collapse

Scholars, commentators, and practitioners have written trust and safety’s obituary.
For Kate Klonick, the “golden age” of content moderation and efforts to create healthy
digital environments is over.168 This conclusion is certainly reasonable: There is
significant evidence of trust and safety’s collapse apparently due to budgetary
retrenchment, macroeconomic uncertainty, and financial instability.

Although the evidence of collapse predated 2022, it was undeniable at that point. Elon
Musk led the way with personnel changes. Immediately upon acquiring Twitter in
October 2022, Musk fired Vijaya Gadde, who oversaw all things trust and safety in her
role as Chief of Legal, Policy, and Trust. 169 Roth, who took over for Gadde, resigned a
month later; cyber mobs inflamed by Musk viciously attacked Roth online, resulting in
his leaving his home for fear for his safety. 170 In December 2022, Musk shut down the
Trust and Safety Council an hour before the Council’s arranged online meeting with
executives to discuss next steps. 171 X reduced the trust and safety team from 230 full-time

168 Klonick, supra note 17.

169 KATE CONGER & RYAN MAC, CHARACTER LIMIT: HOW ELON MUSK DESTROYED TWITTER 208, 263 (2024).
170 Donie O’Sullivan, Former Top Twitter Official Forced to Leave Home Due to Threats Amid ‘Twitter Files’

Release, CNN (Dec. 12, 2022), https://www.cnn.com/2022/12/12/tech/twitter-files-yoel-roth. Ella Irwin,

who took over Roth’s role, resigned on June 1, 2023; the role remained vacant for nine months until Kylie
McRoberts stepped in to lead the much narrower project of “increase[ing] transparency through labels”
and “improv[ing] security with passkeys.” Ashley Belanger, X’s New Head of Safety Must Toe Elon Musk’s
Line Where Others Failed, ARS TECHNICA (Apr. 3, 2024), https://arstechnica.com/tech-policy/2024/04/xs-
new-head-of-safety-must-toe-elon-musks-line-where-others-failed.
171 Cat Zakrzewski, Joseph Menn & Naomi Nix, Twitter Dissolves Trust and Safety Council, WASH. POST (Dec.

12, 2022), https://www.washingtonpost.com/technology/2022/12/12/musk-twitter-harass-yoel-roth/.

23
staff to 20.172 80% of engineers working on trust and safety were fired. 173 Contractors
working on content moderation were cut by 43%. 174

Other leading tech companies followed suit. Snap shrunk its trust and safety
personnel by 27%, from a peak of 3,051 full-time and contract workers in 2021 to 2,226 in
2023. 175 Meta’s layoffs of 21,000 full-time jobs “had an outsized effect on the company’s
trust and safety work.” 176 Meta fired “at least 16 members of Instagram’s well-being
group” and terminated “more than 100 positions related to trust, integrity, and
responsibility.” 177 Hundreds of contractors working on content moderation were fired.178
Google laid off one-third of employees in a unit tackling misinformation, radicalization,
and toxicity. 179 The company made further cuts to trust and safety staff in 2024. 180 Twitch
fired 15% of trust and safety employees, including key individuals addressing “physical
threats, violence, terrorism groups, and self-harm.” 181 Those cuts “caused a big hit to
institutional knowledge.” 182 One of us (Citron) was working on intimate privacy issues
with Twitch trust and safety employees from late 2023 until May 2024 when they were

172 Ethan Zuckerman, When the Internet Becomes Unknowable, PROSPECT MAG. (Nov. 1, 2023),
https://www.prospectmagazine.co.uk/ideas/technology/63752/when-internet-becomes-unknowable-
social-media-tools.
173 Thomas Brewster, Musk’s X Fired 80% Of Engineers Working On Trust And Safety, Australian Government

Says, FORBES (Jan. 10, 2024), https://www.forbes.com/sites/thomasbrewster/2024/01/10/elon-musk-

fired-80-per-cent-of-twitter-x-engineers-working-on-trust-and-safety/.
174 Ben Goggin, Big Tech Companies Reveal Trust and Safety Cuts in Disclosures to Senate Judiciary Committee,

NBC News (Mar. 29, 2024), https://www.nbcnews.com/tech/tech-news/big-tech-companies-reveal-

trust-safety-cuts-disclosures-senate-judicia-rcna145435.
175 Id.
176 Hayden Field & Jonathan Vanian, Tech Layoffs Ravage the Teams that Fight Online Misinformation and Hate

Speech, CNBC (May 26, 2023), https://www.cnbc.com/2023/05/26/tech-companies-are-laying-off-their-

ethics-and-safety-teams-.html.
177 Id.
178 Malea Martin & Cameron Rebosio, Terminated Meta Content Moderators Worry About Fake News Flourishing

in their Absence, ALMANAC NEWS (Jan. 25, 2023),

https://www.almanacnews.com/news/2023/01/25/terminated-meta-content-moderators-worry-about-
fake-news-flourishing-in-their-absence/; Max Zahn, Sales Slump at Facebook Parent Meta, Stock Tumbles, ABC
NEWS (Oct. 26, 2022), https://abcnews.go.com/Business/facebook-parent-meta-expected-report-sales-
slump-amid/story (explaining that declining ad sales and competition from TikTok as the reason for
Meta’s slump, including Apple’s iOS privacy updates limiting online ad targeting capabilities.)
179 Field & Vanian, supra note 176.
180 Davey Alba & Shirin Ghaffary, Google Trims Jobs in Trust and Safety While Others Work ‘Around the Clock,’

BLOOMBERG (Mar. 1, 2024), https://www.bloomberg.com/news/articles/2024-03-01/google-trims-jobs-

in-trust-and-safety-while-others-work-around-the-clock.
181 Hayden Field, Twitch Terminates All Members of Its Safety Advisory Council, CNBC (May 30, 2024),

https://www.cnbc.com/2024/05/30/twitch-ends-contracts-for-its-safety-advisory-council-sac-trust-
safety.html; Safety Advisory Council, Twitch Safety, https://safety.twitch.tv/s/article/Safety-Advisory-
Council?language=en_US.
182 Field & Vanian, supra note 176.

24
fired. Twitch terminated the contracts of all nine members of its Safety Advisory
Council. 183

In addition to personnel dismissals, companies changed their speech policies to

tolerate more abusive content and removed safety features. 184 X stopped addressing hate
speech (in English) and reinstated previously suspended accounts, including those held
by known neo-Nazis. 185 The company disabled features that allowed users to report
election disinformation for assessment and possible removal. 186 YouTube instructed
content moderators to leave up videos, even if they featured prohibited slurs or
misinformation, if the videos could be considered in the public interest. 187

The most significant retreat from trust and safety was announced on January 7, 2025.
That day, Mark Zuckerberg released a video describing wide-ranging cuts and
substantive changes to Meta’s trust and safety efforts. First, Meta was ending its fact-
checking program in the United States and replacing it with a community notes system
in which users would append information to posts in a manner “similar to X.” 188 Second,
Meta was removing restrictions on dehumanizing speech about women, transgender
people, gay, bi, and queer individuals, and immigrants. 189 A leaked training manual
provided examples of newly permissible speech on Facebook and Instagram: “Gays are
freaks; “Immigrants are grubby, filthy pieces of shit;” and “Trans people are immoral.”190
The updated rules explained that posts comparing women to “household objects or

183 Id.
184 Jon Brodkin, “Yeah, They’re Gone”: Musk Confirms Cuts to X’s Election Integrity Team, ARS TECHNICA (Sept.
28, 2023), https://arstechnica.com/tech-policy/2023/09/musk-slashes-x-election-integrity-group-claims-
they-undermined-elections/.
185 Id.; David Klepper, Report: Tweets with racial slurs soar since Musk takeover, AP NEWS (Nov. 10, 2022); Hate

Speech on X Surged for at Least Eight Months After Elon Musk Takeover—New Research, THE CONVERSATION
(Feb. 12, 2025), https://theconversation.com/hate-speech-on-x-surged-for-at-least-8-months-after-elon-
musk-takeover-new-research-249603.
186 Nora Benavidez, Big Tech Backslide: How Social-Media Rollbacks Endanger Democracy Ahead of the 2024

Elections, FREE PRESS REPORT (Dec. 2023), https://www.freepress.net/sites/default/files/2023-

12/free_press_report_big_tech_backslide.pdf.
187 Nico Grant & Tripp Mickle, YouTube Loosens Rules Guiding the Moderation of Videos, N.Y. TIMES (June 9,

2025). Before the change, YouTube instructed content moderators to remove public interest related videos
if a “quarter of content broke the platform’s rules.” Id. After the change, YouTube’s trust and safety officials
advised moderators that “half a video could break YouTube’s rules and stay online.” Id. Staff were told to
leave up a video that contained a slur about a transgender person because the 43-minute video had “only
a single violation of the platform’s harassment rule forbidding ‘malicious expression against an identifiable
individual.” Id.
188 Justin Hendrix, Transcript: Mark Zuckerberg Announces Major Changes to Meta’s Content Moderation

Policies and Operations, TECH POLICY PRESS (Jan. 7, 2025), https://www.techpolicy.press/transcript-mark-

zuckerberg-announces-major-changes-to-metas-content-moderation-policies-and-operations/.
189 Id.
190 Sam Biddle, Leaked Meta Rules: Users are Free to Post “Mexican Immigrants are Trash!” or “Trans People are

Immoral,” THE INTERCEPT (Jan. 9, 2025), https://theintercept.com/2025/01/09/facebook-instagram-meta-

hate-speech-content-moderation/.

25
property” were permissible. 191 Third, the company would no longer proactively scan
posts for policy violations except in the most serious cases concerning terrorism, drug
sales, and child sexual exploitation material.192 Meta was also “dismantling a system to
prevent the spread of misinformation.” 193 Meta would stop using machine-learning
classifiers that, it estimated, had reduced the reach of misinformation by more than 90
percent. 194 Misinformation was “eligible for free amplification on Facebook, Instagram,
and Threads on true stories.” 195

Meta shrank trust and safety on its services as well. In late 2022, engineers working
on the misinformation team were poised to release a fact-checking tool that would let
third-party fact-checkers add comments at the top of questionable articles posted on
Facebook. 196 Although the tool had initial buy-in from executives, the effort was
abandoned in early 2023. 197 At the same time, as Meta touted the metaverse, it ignored
reports of sexual harassment and sexual assaults and failed to hired content moderations
for its virtual world. 198

B. Challenging the Narrative of Collapse

No doubt, the biggest tech companies have fired workers, terminated contracts,
changed rules, and contracted holistic trust and safety practices. It would be easy to turn
to economic explanations to explain these moves. Some tech executives had long
bemoaned that they had to spend resources on the “cost center” of trust and safety. 199 But

191 Alexa Corse, Meghan Bobrowsky & Jeff Horwitz, Social-Media Companies Decide Content Moderation Is

Trending Down, WALL ST. J. (Jan. 7, 2025), https://www.wsj.com/tech/social-media-companies-decide-

content-moderation-is-trending-down-.
192 Hendrix, supra note 188.
193 Casey Newton, Meta Just Flipped the Switch that Prevents Misinformation from Spreading in the United States,

PLATFORMER (Jan. 24, 2025), https://www.platformer.news/meta-ends-misinformation-enforcement-

zuckerberg/.
194 Id.
195 Id. These changes did not sit well with Meta’s Safety Advisory Council, though most members hail from

outside the United States and the changes do not apply to their constituencies.
196 Field & Vanian, supra note 176.
197 Id.
198 Danielle Keats Citron, The Continued (In)visibility of Cyber Gender Abuse, 133 YALE L.J. FORUM 333 (2023).

Meta’s then-president for global affairs Nick Clegg attributed its failure to tackle abuse to the fact that
people will inevitably attack people in the metaverse and people will always find ways to misuse
technologies. Id. at 352. Despite reports of sexual harassment and assault, Meta did not hire sufficient
content moderators. Meta’s Chief Tech Officer Andrew Bosworth said that “moderation in the metaverse
‘at any meaningful scale is practically impossible.’” Id. at 353.
199 See, e.g., Approaches to Trust & Safety, TSPA, https://www.tspa.org/curriculum/ts-
fundamentals/industry-overview/ts-approaches/ (last visited Aug. 7, 2025) (noting that executives may
view a centralized approach to trust and safety as a “cost center”); Vittoria Elliott, Big Tech Ditches Trust
and Safety. Now Startups Are Selling It Back As a Service, WIRED (Nov. 6, 2023),
https://www.wired.com/story/trust-and-safety-startups-big-tech/ (quoting Talha Baig, former Meta
engineer and trust-and-safety startup founder, on how executives view trust and safety).

26
to say trust and safety is collapsing for mostly economic reasons is, at best, incomplete.
On the ground events tell a far more complex story.

Personal affront and political maneuvering seemingly played a role as well. In

apparent pursuit of more “masculine energy” at his platforms, 200 Zuckerberg attributed
Meta’s changes to his anger that his services had been “accidentally censoring” millions
of posts a day. 201 (Zuckerberg was no naïve; he knew his company’s algorithms might
under- and over-filter content given the uncertainty created by Section 230’s legal shield.)
One of those posts was his: Zuckerberg got upset after Facebook failed to boost a post
about his ACL surgery; Meta’s algorithms had been tweaked to slow the spread of health-
related content given concerns about health-related misinformation. 202 Now for the
broader political moment: Zuckerberg attributed the changes in trust and safety to the
“cultural tipping point” of the 2024 Presidential election. 203 Echoing the talking points of
Republican lawmakers and Trump, Zuckerberg said that Meta would be relocating trust
and safety and content moderation teams from California to Texas to “remove the
concern that biased employees are overly censoring content” from conservatives. 204 Meta
executives “gave a heads up to Trump officials about the changes in policy;” Zuckerberg
dined with President-elect Trump at Mar-a-Lago in November 2024. 205

Writing in 2023, TSPA Executive Director Charlotte Willner noted: “Our community
has been tested like never before” by widespread layoffs and “a political climate
increasingly hostile to basic trust and safety practices.” 206 Willner was capturing the
pressure building as a result of Republican lawmakers insisting that social media had
been “censoring” conservative speech. 207 “Preserving Free Speech and Reining in Big

200 Alyssa Goldberg, Zuckerberg says companies need more ‘masculine energy.’ What does that even mean?, USA
TODAY, https://www.usatoday.com/story/life/health-wellness/2025/01/17/mark-zuckerberg-meta-
workforce-masculine-energy/.
201 Corse, Bobrowsky, & Horwitz, supra note 191. As a result, Zuckerberg demanded a review of the rule

and potential overreach. Id.

202 Id.
203 Mike Isaac & Theodore Schleifer, Meta to End Fact-Checking Program in Shift Ahead of Trump Term, N.Y.

TIMES (Jan. 7, 2025), https://www.nytimes.com/2025/01/07/technology/meta-fact-checking-

facebook.html.
204 Hendrix, supra note 188.
205 Isaac & Schleifer, supra note203.
206 Id.
207 Danielle Keats Citron & Jonathon Penney, Empowering Speech by Moderating It, 153 DAEDELUS 31 (2024).

There is little proof to support the claim that tech companies are systematically silencing conservative
voices. Mary Anne Franks, FEARLESS SPEECH: BREAKING FREE FROM THE FIRST AMENDMENT (2024); Mary
Anne Franks, Curating Cyberspace: Rights, Responsibilities, and Opportunities, TECH POLICY PRESS (May 14,
2025), https://www.techpolicy.press/curating-cyberspace-rights-responsibilities-and-opportunities/ (“a
handful of massively influential tech platforms that claim to be neutral but have prioritized reactionary
conservative content for years” and a “bevy of conservative social media forums, including one owned by
President Donald Trump himself, that consistently push out far-right, extremist, neo-Confederate
content.”).

27
Tech Censorship” had become the theme of congressional outcries and hearings. 208 State
lawmakers joined the fray. Florida and Texas passed laws restricting social media
companies’ ability to moderate content, though those laws were enjoined on First
Amendment grounds. 209 President Trump sued Meta for banning his account in the
aftermath of the January 6 attack on the U.S. Capitol. All of this signaled to tech executives
that they did not need to, nor should, spend resources on trust and safety.

At its core, though, the practice of trust and safety is changing because the law of trust
and safety has been changing. Whereas in the United States there has been uncertainty in
the wake of Section 230’s passage and broad interpretation, in the European Union, new
laws have catalyzed a shift toward recordkeeping, transparency, and procedural
compliance in a public-private partnership between industry and government. 210 For
good reason, tech companies have made clear that their shrinking of trust and safety in
the United States does not apply in the European Union and other nations. 211 The EU
Digital Services Act of 2022 (DSA) creates considerable legal responsibilities for the
biggest online platforms, including regular risk assessments, and imposes financial
penalties of up to 6% of annual revenue. It requires content platforms to publish annual
transparency reports on content restrictions, government requests for user data, and the
use of automated moderation tools.212 Under the DSA, many platforms are required to
provide information about their algorithms and languages spoken by their content
moderators. In 2023, the European Commission launched the DSA Transparency
Database. The United Kingdom and other nations have similarly regulated online
platforms. 213

208 Preserving Free Speech and Reining in Big Tech Censorship, U.S. House of Representatives, 118th Cong.,
1st sess. (Mar, 28, 2023); Representatives Cathy McMorris Rodgers, Robert E. Latta, Gus M. Bikirakis, Letter
to President Biden (Sept. 13, 2022) (raising concerns that President Biden pressured Twitter and Facebook
to “censor speech or silence individuals with whom you disagree,” including news about the Hunter Biden
laptop). One of us (Waldman) testified before Congress at one of these hearings, demonstrating the lack of
evidence for Republicans’ claims. See Filtering Practices of Social Media Platforms, U.S. House of
Representatives Comm. on the Judiciary, 115th Cong., 2d sess. (Apr. 26, 2018).
209 Citron & Penney, supra note 207, at 33. In Moody v. Netchoice, the Supreme Court affirmed the notion that

tech companies’ speech policies and decisions constitute editorial choices protected by the First
Amendment. Moody v. Netchoice, 603 U.S. 707 (2024).
210 See Margot Kaminski, Binary Governance: Lessons from the GDPR’s Approach to Algorithmic Accountability,

92 S. CAL. L. REV. 1529 (2019) (citing the literature and describing collaborative or “new” governance as
involving public private partnerships in which the compliance work is done by regulated entities
themselves).
211 Amandine Hess, Do Meta’s Announcements Run Counter to the European Regulation on Digital Services?,

EURONEWS.COM (Jan. 10, 2025), https://www.euronews.com/next/2025/01/10/do-metas-

announcements-run-counter-to-the-european-regulation-on-digital-services.
212 2025 TRUST AND SAFETY UK: STATE OF THE INDUSTRY REPORT 6 (Mar. 25-26, 2025),
https://www.iqpc.com/events-trust-and-safety-summit/downloads/2025-trust-safety-state-of-industry-
report.
213 UK Online Safety Act—2025 and Beyond, INTERNET COMM’N BLOG (Jan. 31, 2025),
https://www.inetco.org/news/uk-online-safety-act-2025-and-beyond (discussing the UK Online Safety
Act of 2023).

28
 At the second UK Trust and Safety Summit held in March 2025, panelists focused on
regulatory compliance and “readiness for new laws” that address platform safety.214
Company representatives discussed their approaches to risk assessment, the need to
tackle different regulatory requirements across jurisdictions, and “shared principles
beyond regulation.” 215 For this reason, the UK Trust and Safety Summit’s 2025 report
estimated that the global trust and safety technology market would increase by ten billion
dollars by 2030 in comparison to 2022. 216 Given “stringent regulations” and “urgent and
unmet company needs,” there has been a near tripling of the amount of venture
investment in the global trust and safety sector.217 As Pitchbook explained, trust and
safety startups raised $324 million globally in 2023, up from $48.5 million in 2019. 218

Compliance management, risk mitigation, user retention, greater advertising revenue,

and a shift to automated compliance vendors are behind that projected growth. A study
by the UK’s Department of Science, Innovation & Technology explained that platforms
are increasing their investment in trust and safety vendor products. 219 These vendors are
now reaping the benefits of the social practice of tech company mimicry and knowledge
sharing: As Lucas Wright has shown, companies are now looking to each other and to
these competitors’ vendors as a means of complying with the DSA. 220

This trend to procedural compliance should come as no surprise to legal scholars. For
better or for worse, proceduralist compliance is the dominant form of governance practice
in neoliberalism. Julie Cohen has demonstrated that the legal institutions of informational
capitalism as a whole are “managerial,” relying on audits, third-party vendors, and
procedural compliance tools as public regulatory institutions are hollowed out.221
Financial regulation after the 2008 Financial Crisis also relies on audits, organizational
structures, and compliance vendors. 222 Privacy and data protection practices inside tech

214 Trust & Safety Summit 2025: A Desire to do the Right Thing in a Sector Defying its Challenges (Apr. 17,
2025), https://www.inetco.org/news/trust-safety-summit-2025-a-desire-to-do-the-right-thing-in-a-
sector-defying-its-challenges.
215 Id.
216 STATE OF THE INDUSTRY REPORT, supra note 212, at 10.
217 Id. at 12.
218 Rosie Bradbury, Meet the Cyber Trust and Safety Startups that Have Raised the Most Venture Capital,

PITCHBOOK (May 29, 2024), https://pitchbook.com/news/articles/top-trust-safety-startups-capital-raised

(explaining that AI-enabled tools that scan and identify sources of misinformation or scan content for abuse
have attracted considerable funding).
219 STATE OF THE INDUSTRY REPORT, supra note 212, at 10.
220 Wright, supra note 32, at *18.
221 COHEN, supra note 37, at 144-145, 192-193; see also Julie E. Cohen & Ari Ezra Waldman, Introduction:

Framing Regulatory Managerialism as an Object of Study and Strategic Displacement, 86 LAW & CONTEMP. PROBS.
i, ii-iv (2023) (defining managerialism and regulatory managerialism).
222 Rory Van Loo, The New Gatekeepers: Private Firms as Public Enforcers, 106 VA. L. REV. 467, 485-86 (2020)

(demonstrating how CFPB regulators outsource regulation of third parties to banks); Rory Van Loo,
Regulatory Monitors: Policing Firms in the Compliance Era, 119 COLUM. L. REV. 369, 397-98 (2019) (describing

29
companies focus on “comprehensive privacy programs,” data protection officers, record
keeping, and transparency, among other proceduralist practices. 223 And corporate anti-
discrimination practices have also become policy statements, diversity offices, bias
training, and internal appeals, replacing substance with procedure. 224 That trust and
safety has followed the same trend is unremarkable.

As a normative matter, however, the evolution of trust and safety highlights its
precarity, even from its earliest days, but even more so today. But the shift to compliance
raises unique challenges, to which we now turn.

III. THE PAST AND FUTURE OF TRUST AND SAFETY

This Part takes stock of the potential effects of trust and safety’s shift from a mostly
self-governed system of rules and design to a public-private partnership focused mostly
procedural compliance. We explore the consequences in two clusters. First, we consider
the problem structurally and identify inherent inconsistencies and weaknesses in the
compliance-first approach. Second, we explore whether the compliance turn is up to the
challenge of addressing the harms holistic trust and safety was supposed to address in
the first place. The jury is still out, but hope remains.

A. The Inherent Risks of a Compliance Model for Trust and Safety

The shift to a compliance is not neutral. It is changing the face of trust and safety, and
not necessarily for the better. In fact, given the proliferation of the compliance model of
governance throughout the legal machinery of informational capitalism, there is reason
to worry that the compliance alternative will be worse than even a wildly inadequate
holistic trust and safety approach.

Compliance practices are expensive. Traditionally, they require in-house staff, legal
input, substantial time, and set up costs. Under the DSA, companies have to keep records
that they likely never kept before. And despite the specificity of its compliance mandates,
even the DSA lacks clarity at times. The largest, wealthiest, and most established interests
have the closest connections with regulators in most jurisdictions, giving them access to
helpful feedback when crafting their regulatory programs. As a result, dominant players
will be able to act first, establish new compliance practices on their own, and elevate their
own plans as paradigmatic for the entire industry. Therefore, the compliance turn will

the role of internal compliance departments in financial regulation as a form of “collaborative

governance”).
223 WALDMAN, supra note 34, at 130-142; Waldman, False Promise, supra note 27, at 803-815; Ari Ezra

Waldman, Privacy, Practice, and Performance, 110 CALIF. L. REV. 1221 (2022).
224 EDELMAN, WORKING LAW, supra note 53.

30
not solve the problem that trust and safety was ultimately determined by what the
biggest social media platforms wanted it to be. 225

By contrast, smaller companies will be forced to outsource much of their trust and
safety compliance to outside vendors. 226 Many trust and safety vendors use the software-
as-a-service model whereby they promise platforms that they can use advanced
technology (often clustered together under the heading of “artificial intelligence”) to
automate compliance with the DSA and other trust and safety laws. 227 Their goal is
nothing short of “own[ing] the platform through which the market for governance and
compliance” of trust and safety operates. That was dangerous during trust and safety’s
holistic stage, where achieving healthy digital environments was necessarily a human-
centered, iterative process with technology serving a supporting role. Proceduralist
compliance, with its emphasis on record keeping, reports, and transparency, is
structurally (though still imperfectly) suited to vendor automation. This will hasten the
outsourcing of trust and safety and the resulting erasure of on-the-ground expertise. As
we lose the creativity of human beings, the shift to compliance may facilitate the
overreliance on automated technologies to solve problems they are ill equipped to
handle.

The turn to compliance and procedure shifts the kind of organizational learning that
will take place throughout the trust and safety profession. Whereas previously, trust and
safety journals, conferences, meetings, and collaborations featured healthy mixes of new
practices, harm use cases, policy matters, and compliance, among other things, the shift
to proceduralism will likely shift knowledge production. Rather than focusing on
strategies to make a platform safer, compliance professionals focus their energies on
identifying strategies to comply with the law. Those questions could conceivably have
the same answer, but the former is steeped in substantive results whereas the latter lends
itself to creative lawyering and avoidance.

Proceduralist compliance also elides the very substantive harms that holistic trust and
safety was at least supposed to combat. Trust and safety emerged in reaction to gendered
cyber abuse, among other harms like fraud, defamation, and misinformation. But

225 This problem is not unique to social media trust and safety. See Eric A. Posner, Law, Economics, and
Inefficient Norms, 144 U. PA. L. REV. 1697, 1727 (1996) (“Highly unequal endowments of group members
may be evidence of inefficient norms. The more powerful members may prefer and enforce norms that
redistribute wealth to them, even when those norms are inefficient.”); Lloyd L. Weinreb, Custom, Law and
Public Policy: The INS Case as an Example for Intellectual Property, 78 VA. L. REV. 141, 146-47 (1992) (noting
that “the better financed private interest” will win, “rather than a careful, systematic” rule that “will serve
the community as a whole”).
226 One of us (Waldman) discussed how this plays out in the privacy context. See Ari Ezra Waldman,

Outsourcing Privacy, 96 NOTRE DAME L. REV. REFLECTION 194, 196 (2021); see also Rohan Grover, Dissertation:
Consent Theater: A Sociotechnical Analysis of Data Privacy Law in Action, University of Southern
California Annenberg School of Communications (2025) (manuscript on file with authors).
227 Wright, supra note 32, at *29.

31
procedural approaches arguably offer “no framework for thinking systematically about
the interrelationships between political and economic power.” 228 Compliance practices,
transparency reports, and impact assessments are supposed to keep trust and safety
concerns front of mind throughout the company. But the filter of paperwork makes it
even more difficult than before to stop the very harms that encouraged companies to put
their policies down on paper in the first place.

To be sure the DSA leaves room for European regulators to focus on substantive
issues. X is being investigated for promoting illegal content (including hate speech,
terrorist and violent content, and disinformation), failure to tackle dark patterns, and
inadequate compliance with transparency requirements. 229 The European Commission
has also sent requests for information to YouTube, Snapchat, and TikTok under the DSA,
focusing on the design and functioning of recommender systems. 230 Under the DSA,
platforms must “assess and adequately mitigate risks stemming from recommender
systems, including risks to mental health of users and the dissemination of harmful
content arising from the engagement-based design of these algorithms.” 231 YouTube and
Snapchat have been asked to address “their role in amplifying certain systemic risks,
including those related to the electoral process and civic discourse, users’ mental well-
being (e.g. addictive behavior and content ‘rabbit holes’), and the protection of
minors.” 232

But we have seen this episode before, and we do not like it. 233 Legal endogeneity
theory posits that procedural tools of compliance independent of substantive outcomes
can stand in for actual adherence to the law. 234 Therefore, it is not difficult to imagine
these investigations settling—as so many Federal Trade Commission investigations have
settled in the United States 235—with small fines and minor procedural changes that do
little to encourage more comprehensive trust and safety practices in the future. Only time
will tell how the EU will make use of the tools it has in the DSA.

228 Britton-Purdy, Grewal, Kapczynski & Rahman, supra note 35, at 1790.
229 Ramsha Jahangir, Understanding the EU’s Digital Services Act Enforcement Against X, TECH POLICY PRESS
(Apr. 5, 2025), https://www.techpolicy.press/understanding-the-eus-digital-services-act-enforcement-
against-x/; Jess Weatherbed, EU Races to Conclude Investigation into X’s Content Moderation Efforts,
VERGE (Jan. 8, 2025), https://www.theverge.com/2025/1/8/24338985/x-content-moderation-eu-probe-
dsa-rules.
230 Press Release, European Commission Sends Requests for Information to YouTube, Snapchat, and TikTok

on Recommender Systems Under the Digital Services Act (Oct. 2, 2024), https://digital-
strategy.ec.europa.eu/en/news/commission-sends-requests-information-youtube-snapchat-and-tiktok-
recommender-systems-under-digital.
231 Id. (emphasis added).
232 Id.
233 With apologies to Taylor Swift. See Taylor Swift, The Tortured Poets Department, in The Tortured Poets

Department Anthology (Apr. 19, 2024) (“But I've seen this episode and still loved the show”).
234 EDELMAN, WORKING LAW, supra note 53.
235 Waldman, Performance, supra note 223, at 1233-1241.

32
B. The Dangers Awaiting Compliance

If the compliance turn for trust and safety does weaken an already inadequate form
of platform governance, it will be unable to address the profound and undeniable harms
faced by users. Cyberstalking, harassment, threats, doxing, nonconsensual intimate
images (real or synthetic), foreign malign influence campaigns, election disinformation,
and hate speech: All damage people’s careers, education, safety, free expression, and
other life opportunities. As our scholarship has explored, women, LGBTQ+ people, racial
and religious minorities, immigrants, disabled people, and often people with intersecting
marginalized identities shoulder the brunt of the abuse. 236

We do not have a precise accounting of the abuse and their harms due to the
ratcheting back of trust and safety, but what we know is troubling. Facebook “has been
overrun by AI-generated spam;” “nonconsensual intimate imagery is easy to find on both
Facebook and Instagram. 237 According to a 2024 study, nonconsensual intimate imagery
remained on X for weeks after being reported. 238

The risk for young people should be obvious. Of Meta’s retreat from proactively
filtering bullying, harassment, and self-harm content, Arturo Bejar, a former Meta
engineering director, noted: “I shudder to think what these changes will mean for our
youth, Meta is abdicating their responsibility to safety, and we won’t know the impact of
these changes because Meta refuses to be transparent about the harms teenagers
experience, and they go to extraordinary lengths to dilute or stop legislation that could
help.” 239 It is also particularly troubling for young people that pro-eating disorder content
has made a comeback on X where there is an “unending stream of extreme dieting
instructions” and other content glorifying thinness. 240 X appears to be actively

236 See, e.g., CITRON, HATE CRIMES IN CYBERSPACE, supra note 2, at 13-19; CITRON, FIGHT FOR PRIVACY, supra
note 2, at 39-40; Ari Ezra Waldman & Mary Anne Franks, Sex, Lies, and Videotape: Deep Fakes and the Free
Speech Delusion, 78 MD. L. REV. 892 (2019); Ari Ezra Waldman, A Breach of Trust: Fighting Nonconsensual
Pornography, 102 IOWA L. REV. 709 (2019); Ari Ezra Waldman, Triggering Tinker: Student Speech in the Age of
Cyberharassment, 71 U. MIAMI L. REV. 427 (2016); Danielle Keats Citron & Mary Anne Franks, Criminalizing
Revenge Porn, 49 WAKE FOREST L. REV. 345, 353-354 (2014); Danielle Keats Citron, Cyber Civil Rights, 89 B.U.
L. REV. 61, 69-81 (2009).
237 Jason Koebler, Has Facebook Stopped Trying?, 404 MEDIA (June 24, 2024), https://www.404media.co/has-

facebook-stopped-trying/.
238 Prithvi Iyer, New Research Highlights X’s Failures in Removing Non-Consensual Intimate Media, TECH. POLICY

PRESS (Oct. 11, 2024), https://www.techpolicy.press/new-research-highlights-xs-failures-in-removing-

nonconsensual-intimate-media/.
239 Barbara Ortutay, Meta rolls back hate speech rules as Zuckerberg cites ‘recent elections’ as a catalyst, AP NEWS

(Jan. 8, 2025), https://apnews.com/article/meta-facebook-hate-speech-trump-immigrant-transgender-

41191638cd7c720b950c05f9395a2b49.
240 Kaitlyn Tiffany, The Pro-Eating Disorder Internet Is Back, ATLANTIC (Dec. 18, 2024),
https://www.theatlantic.com/technology/archive/2024/12/eating-disorder-content-x/681036/.

33
recommending eating-disorder material and pro-eating disorder groups have thousands
of members. 241

The knowledge and transparency growing out of collaborative efforts are being lost
as well. In March 2023, X announced that research access to the API, which was free,
would cost $42,000 a month, “putting it out of the reach for most researchers.” 242 The
“thousands of people” using Twitter’s API to study election interference and other online
pathologies could not continue their research. 243 Other social media platforms popular
with researchers (like Reddit) followed suit, charging researchers and journalists to access
posts. 244 So too with Meta: academics, journalists, and other interested parties have been
shut out of “CrowdTangle, a free service showing what posts on Facebook and Instagram
were most popular via searchable real-time dashboards.” 245 Meta directed users to a less
useful alternative that lacks Crowdtangle’s core features. 246 Without such insight,
advocacy groups will have less proof to offer the public and lawmakers—it is far easier
to dismiss online abuse as one-offs rather than an endemic problem.

And to boot researchers are now facing lawsuits for their studies of online activities.
In July 2023, X sued the non-profit Center for Countering Digital Hate (CCDH), which
studies the spread of extremist speech online. 247 CCDH reported that hate speech
targeting minority communities increased since Musk acquired X. 248 X’s lawsuit alleges
that those findings are false and thus have hurt their business, which is an obvious effort
to intimate researchers into silence. As Ethan Zuckerman warns, “increasing
disinformation on some platforms, the closure of tools used to study social media,
lawsuits against investigations on disinformation” poses an “uphill battle to understand
what happens in the digital public sphere in the near future.” 249 Also unsurprisingly X
has withdrawn from other collaborations, including the work of Christchurch Call. 250

C. Possible Responses

Holistic trust and safety in the United States may well rise again, but it will be
different. The rise, spread, and evolution of trust and safety in today’s political climate
offers a unique opportunity for scholars, policymakers, and practitioners to build anew,
to imagine what trust and safety—if we continue to call it that—might look like in the

241 Id.
242 Zuckerman, supra note 172.
243 Benson, supra note 114.
244 Id.
245 Newton, supra note 18.
246 Zuckerman, supra note 172.
247 Id.
248 Id.
249 Id.
250 Paresh Dave, Inside Two Years of Turmoil at Big Tech’s Anti-Terrorism Group, WIRED (Sept. 30, 2024, 5:00

AM), https://www.wired.com/story/gifct-x-meta-youtube-microsoft-anti-terrorism-big-tech-turmoil/.

34
future. In the short term, the profession will be flexing its muscles abroad, but in starkly
different ways. Whether we see effective push back in the United States depends on a
number of factors.

Information and engineering scholars Rachel Moran, Joseph Shafer, Mery Bayar, and
Kate Starbird have called for strengthening trust and safety as a profession and as an
industry, pushing back against misconceptions among the public and politicians, and
building up significant research agendas among scholars to arm trust and safety
personnel with the weapons they need to push their bosses into action. 251 These
recommendations make sense, but they highlight the fragility and precarity in which
trust and safety has always found itself. Even at its best, the field has always had to
navigate between a singular vision of corporate profits tied to engagement, which is
inherently anathematic to moderation, and the actual horrors of abuse online. For Moran
and her colleagues to rightly call on researchers to help create an arsenal of data for trust
and safety workers to use in meetings with decision-makers is to also portray trust and
safety professionals as modern-day Oliver Twists, asking, with bowl in hand, for some
more.

But we do not have rely exclusively on the resilience of those working in trust and
safety. Nor are we limited to a legal regime in which safety for users depends on someone
else begging for it. There are steps for law to take.

U.S. policymakers should consider reform to Section 230. One of us (Citron) has been
working with Massachusetts Congressman Jake Auchincloss on reasonable changes to
the law that created a self-governing regime so easily thwarted and diverted. Among
other things, the bill would deny immunity to those in the business of cyber gender
abuse. 252 It would require platforms to have effective reporting structures, be designed
such that companies can prevent cyberstalking, intimate-privacy violations, and digital
forgeries and remove nonconsensual intimate imagery. 253 Although by no means a
panacea, these changes are both technological feasible and good for business. They also
avoid the problem of proceduralism taking the place of substance because compliance
would require not just process but also removal, blocking, and prevention.

At the same time, the trust and safety profession needs more protection from the
whims of their employers. One of us (Waldman) has described in detail how the precarity
of privacy lawyers, privacy professionals, and privacy engineers inside tech companies
has prevented these individuals and groups from aggressive advocacy inside their
organizations and forced them to put corporate profits over their interpretations of the

251 Moran et al., supra note 19.

252 Citron, Continued (In)visibility, supra note 198, at 365-366.
253 Id. The bill, in its current form, is called the Intimate Privacy Protection Act. See H.R. 9178, 118th Cong.,

1st sess., https://www.congress.gov/bill/118th-congress/house-bill/9187. Dr. Mary Anne Franks, CCRI’s

President, and Citron continue to work with Representative Auchincloss on the bill.

35
law and their hopes for privacy protective designs. 254 An independent, grassroots union
of privacy professionals that would help prospective tech industry employees negotiate
job security could help ameliorate some of the worst influences of the profit-at-all-costs
mission. 255 A similar worker-powered union could help strengthen the position of trust
and safety professionals.

CONCLUSION

The field of trust and safety was always in a precarious position, earnestly wanting to
do something about unchecked abuse online but having to do it through the filter of
independent and sometimes conflicting corporate motives. The self-regulatory regime
created by Section 230 and spread throughout the ecosystem of tech companies has
shifted to a largely proceduralist regime less open to collaboration, organizational
learning, and on-the-ground human expertise. Instead, trust and safety may be headed
in the direction of paperwork and automation—a regrettable state of affairs that may
hardly help prevent and minimize online harms.

If the field can evolve once, it can evolve again. Trust and safety professionals are
going nowhere. Their world is not collapsing. Scholars are bringing more nuanced
approaches to researching the changes in the field. Policymakers tired of partisan and
disingenuous rhetoric about social media censorship are laying the groundwork for
future legal reforms that could trigger yet more change in trust and safety. In the interim,
some platforms may drift beyond repair and lose the public interest. The ones that remain
will, we hope, be stronger for their long-time commitments to healthy digital
environments.

254 WALDMAN, supra note 34.

255 Id. at 240.

36