
This pair of articles is for product managers who might deal with content moderation issues. We’re not going too deeply into the hot debate about moderation at the major platforms. There’s no shortage of folks throwing in on that. Though we’ll dive in somewhat to take away lessons. We’re going to look more at what’s going on for those who are practitioners as managers and builders, not observers.
Rising Volume of Posts and Debate
Besides the top blogworthy targets of debate, there’s tens of millions of sites with some form of user generated content. (Estimate based on Netcraft’s 2024 Web Server survey. Over a billion websites across 270 million domains. If we look at categories of sites that likely have UGC, it’s maybe 40% Even just half of that is still 50 million.)
This means many of us who don’t happen to be Facebook, Twitter, YouTube, Reddit, etc. have the benefit of not being hauled before Congress to testify on our content moderation policies. However we still must deal with the issues. Especially since manipulation isn’t just about politics, it’s serious dollars. Among the statistics CrowdRiff reports, they include the GrandView Research estimate of 2023 UGC market at $5.36 billion, growing at a 29%+ CAGR through 2030.
This rising volume of content moderation debate is both serious and comical. It’s serious in that issues of false or harmful content are sensibly of great concern. It’s comical in that the volume of postings across media types, (billions per day), has grown to such mind-boggling levels, it’s challenging to contemplate. Is it possible for anyone to deal with this well? (And this is assuming agreement on base policies in the first place.) Some of us perhaps work for Facebook or Twitter or others dealing with mass volumes. But mostly not. One interesting thing about unicorn firms is there’s sometimes few external service options for help. When you’re orders of magnitude larger than most, you either have to invent things yourself or support a whole separate mini-industry. (Because few want to build companies to handle that scale that have such singular customer concentration risk.) Early in any industry, there’s little ecosystem support. We saw this with the early internet as basic web hosting grew, as advertising technology arose and so on. This is also true of content moderation. And while we may be at a breaking point for current methods for the major services, there are still services available for lower volumes.
So when I read Paul Barrett’s views on contract moderation “It’s Past Time to Take Social Media Content Moderation In-House,” it was easy for me to disagree with his conclusions. If moderation goes all in house, it would be an even bigger management headache. Using external firms is more flexible and allows faster adjustments. Such as now, where services are likely using more AI and community based tools. Not every smaller or mid-size firm has the bandwidth to enable such tools in house. Especially if the UGC part of the business is more ancillary than core.
AI is not a panacea. But if billions of posts can be managed accurately via AI, that’d be great, right? There’s challenges of course. One is AI model training. Training AIs for many tasks involves labeling content. Some of this can be unsupervised with machines alone, but other training likely involves at least some form of humans in the loop. Which means we’re still talking about a lot of people being exposed to some ugly things and then labeling them. Note that AI in this case is not simply a linear thing. Logistic regression or neural networks need to classify content types, (such as “bias,” or “some other illegal thing”), into classes first and then judge if it crosses some threshold. So separate models run. Alternatively, there can be a more complex generalized multi-class model. Either way, just as when using GPTs, there’s going to be costs-per-inference here, or similar costs. (Likely increasing if you’re also doing multi-modal content types. And if you’re allowing image uploads, you have this issue.)
Even if your AI training is good, you can’t get away from the basics of precision vs. recall. The higher you set a threshold to find bad things, the more good things will be false positives. Depending on volume, that can be a lot of upset customers complaining about blocked posts and banned accounts. So you have to consider the cost/value of blocking. For illegal content, maybe you use a high percentage and take the hit on the complaints about false negatives. For other content, maybe a lower threshold.
If you want to see how the math works for precision vs. recall in AI, you can see my training session on AI metrics; the precision / recall discussion starts at 19:39. Metrics for AI Product Managers – AI PM Community Session #63. (This was done as part of a PMExercises AI Product Manager Training Program which I highly recommend.)
Let’s just take a look at the numbers the big kids are dealing with… Facebook had 3 Billion users as of late 2023 per SproutSocial. If it were a country, it would be the largest in the world. It’s hard to tell exactly how many total engagements there are. But in a 2024 posting, SocialPilot says it’s over 1 billion stories per day. Some years older data from Wishpond when they ‘only’ had 1.4B users claims they had 4.75 billion items shared per day across the various content types. So we can probably just – at least – double that. In other words, possibly approaching around 10 billion postings in some form or another. Per day. In a Jan 10, 2025 interview with Joe Rogan, Joe Rogan Experience #2255 – Mark Zuckerberg, we can hear Mark Zuckerberg talk about some of these issues. Go to 29:50 to hear about this particular challenge. He talks about issues such as precision and recall and setting the levels for classifiers. Now you might say “I wouldn’t mind taking a crack at that problem for a $200B+ net worth.” But here’s the thing, does anyone really think the $1.5 Trillion market cap Meta/Facebook doesn’t want to stop taking #$$% for this challenge? Of course they’d love to solve this. And yet, the best data scientists you could hire haven’t managed to fully solve this yet. (Though everyone is getting better with AI, another irony is AI might be getting better at being bad as well.)
Most of us don’t face this magnitude of challenge. But we can still learn from these lessons.
Humans Can Be… Challenging
Most people, the overwhelming amount of the time, are good, kind souls who generally go about their days just fine. The challenge is that with billions online, if even a small percentage are problematic, that’s still tens of millions. And then there’s also many folks of good conscience who sometimes vigorously disagree. So if we break this down, we really have two basic levels of problem:
- Outright Bad Actors
- They know they’re doing wrong. From simple pranks to bullying to criminals, terrorists and state-sponsored influence; individual or organized. The good news is that, though there may still be challenges dealing with such activities, there’s usually fairly bright lines here.
- They know they’re doing wrong. From simple pranks to bullying to criminals, terrorists and state-sponsored influence; individual or organized. The good news is that, though there may still be challenges dealing with such activities, there’s usually fairly bright lines here.
- Heated Legitimate Debate
- Across topics, there are those who flame out with their passions for their positions. Obviously politics features prominently here. This sort of thing is arguably more challenging to judge.
We can have differing policies and methods for dealing with such things. But the rise in temperature lately has often been about legitimate debate; the concerns being the usual about truthfulness, harm and so on. Of course, (again refer to Barrett’s article among others), we can look at politics just as an example. We see that both Democrats and Republicans have accused social media companies of bias. It’s possible both are true, given moderation may have been done with different biases on different posts. However, the arguing parties claim systemic and pervasive one-sidedness. This should not be surprising given that we all see through our own perceptual filters. And yet, social media companies can’t just ignore this ironic foolishness given the people complaining have the power to make laws; including crushing Section 230 of the CDD, which we’ll get to.
I can understand Barrett’s view. He believes taking this in house could lead to better training and supervision. He’s probably right. What’s missing though, is the idea of the costs this would incur, as this would be a lot of bodies and liability to onboard. Look who is doing moderation now. These are not dream jobs. They’re going to be lower end and sometimes with minimal training. Now, this is not every firm or every moderator. As with anything else, quality runs the spectrum. And it appears U.S. firms are most costly. Look at the 2020 lawsuit against Facebook. Facebook will pay $52 million in settlement with moderators who developed PTSD on the job. And this is part of the point. It can be a challenging job. See “Inside Facebook’s African Sweatshop,” or “Social Media’s Dark Tasks Outside of Silicon Valley.” So we have emotionally traumatic work being done by poorly paid entry level employees at high volumes. This last story has the challenge partly correct when it says, “Facebook sort of serves as a standard for industrywide problems, concerns, dilemmas. If Facebook can’t solve this problem, what can smaller platforms do? What are other companies doing? Issues of scale are really part of the story.” It’s only partly correct because very few have this volume of challenge.
The Law
Startups will not typically have fully formed Governance, Risk, Compliance (GRC) departments. They might barely have general counsel. So if you’re going go build something here, (or are catching up), it would be good to have at a basic primer. So – at least the U.S. – let’s take a look at what’s involved. The laws generally surround the following large scale areas: general rights, liability, user privacy, and prohibited activities. The 1996 Communications Decency Act (CDA), (and specifically section 230), stands out as a cornerstone of content moderation, alongside laws regulating intellectual property, privacy, and consumer protection. Here’s a quick summary just to have some background. And of course there’s the usual disclaimer: I’m a senior product person who’s built multiple products using UGC, not a lawyer. You need to check any and everything I say with your own legal counsel.
The First Amendment (1791): We’ll start with this big one. You ideally already know the text itself. But whom does it really impact? It limits government restrictions on speech but does not extend to private moderation by businesses. The First Amendment impact is not just about the law itself, or the mass quantity of statutory and case law that’s followed, but a fundamental philosophy that permeates American society; even if perspectives on it differ. Let’s make sure this is really clear because for some reason, some folks like to wave this flag online. You see it in forums all the time, “I have a right to… etc. etc.” The First Amendment simply does not apply to business or individuals in terms of forcing people to allow free speech. I can kick you out of my online service or my conference room or whatever if I don’t like what you have to say. That being said, if you’re perceived as being overly oppressive, people may simply call you out on being overly restrictive and leave.
Communications Decency Act (CDA) Section 230 (1996): Let’s spend some time with this one as it has a lot of impact. It protects online platforms from liability for user-generated content and allows them to moderate content in “good faith.” This is perhaps one of the more controversial items in discussions as to the obligations of services regarding online content.
The CDA itself originally aimed to regulate online indecency, particularly by criminalizing the transmission of obscene or offensive material to minors. Which seems like a laudable enough purpose. However, much of the CDA, including these provisions, was struck down by the Supreme Court in Reno v. ACLU (1997) for violating the First Amendment. Section 230 was not affected and remains in force as the most impactful surviving part of the CDA.
Section 230 in particular, known as the “safe harbor” provision, grants immunity to online platforms from being held liable for user-generated content and allows them to moderate content without being treated as publishers. It is widely regarded as the law that enabled the modern internet by fostering the growth of platforms like social media, forums, and e-commerce sites. And actually, the law specifically classifies these entities as online platforms. They’re not even what you might think of as a distributor, like a bookstore or library. The online platforms are their very own class of thing. They can even moderate without being considered a publisher.
The rationale for Section 230 was to foster innovation and growth of the internet while balancing the need for content moderation. It was designed to address two key issues emerging in the mid-1990s:
- Encouraging Moderation Without Liability: Before Section 230, platforms faced a legal dilemma: if they moderated content, they risked being treated as publishers and held liable for user-generated content; if they didn’t moderate, they risked hosting harmful or offensive material. Section 230 created a safe harbor, allowing platforms to moderate content in “good faith” without assuming legal responsibility for everything posted.
- Promoting Innovation and Free Speech: Lawmakers wanted to enable the development of online platforms by shielding them from excessive lawsuits over user content. This protection incentivized companies to create forums, social media platforms, and other interactive services without the fear of being overwhelmed by litigation.
In summary, Section 230 was intended to strike a balance between enabling platforms to curate their environments and encouraging a vibrant, user-driven internet ecosystem.
Digital Millennium Copyright Act (DMCA) (1998): Requires platforms to remove copyrighted material when notified under its “safe harbor” provisions.
Children’s Online Privacy Protection Act (COPPA) (1998): Imposes restrictions on collecting personal information from children under 13.
Federal Trade Commission (FTC) Act (1914, with amendments): Prohibits deceptive advertising, misleading claims, and unfair business practices in digital content.
California Consumer Privacy Act (CCPA) (2018): Provides privacy rights to California residents, intersecting with content moderation in user data handling.
General Data Protection Regulation (GDPR) (2016, EU): While not a U.S. law, it affects U.S. businesses operating in that regioin by enforcing user privacy rights.
There’s much more of course. But these are the core issues that impact us most directly.
What Can We Do
You’ll have to go to the exciting conclusion for that. I’ll link it in here when it’s done. You’ll have to go to the exciting conclusion for that. I’ll link it in here when it’s done. And… here it is now: Content Moderation: Management Checklist.
Before moving on, consider this overview from CNBC back in 2022…
Why Content Moderation Costs Social Media Companies Billions