Online harm is not an internet problem. It is a power problem – and the internet has given it new reach, new scale, and new tools for deniability.
The harm we describe is the online expression of systems of oppression that predate digital communication: classism, racism, misogyny, Islamophobia, transphobia, ableism, antisemitism, and the targeting of people because of their national origin.
These systems did not emerge online.
They migrated there – and the architecture of online platforms amplified them, accelerated them, and stripped away the friction that might otherwise have constrained them.
The same words cause different harm to different communities. A model built without community knowledge will learn the wrong signals – flagging the vocabulary of a targeted group rather than learning when that vocabulary is being used as a weapon.
This is not a technical limitation. It is a design choice, and it is one we reject.
This principle is supported by the XtremeSpeech research (Maronikolakis et al. 2022), which demonstrates empirically that harm classification cannot generalise across communities without community-specific training data, and by HateCheck (Röttger et al. 2021), which shows that state-of-the-art models systematically misclassify harm directed at some groups more than others.
Individual pieces of content are often ambiguous. Harm frequently manifests as a pattern – coordinated targeting, sustained volume, documented suppression of a community’s ability to speak and participate online.
This principle draws on Susan Benesch’s Dangerous Speech Framework, which establishes that speech becomes dangerous not through its content in isolation but through the relationship between speaker, audience, and target – and on Danielle Citron’s work on coordinated online abuse, which shows that harm to individuals in public life is rarely the product of a single comment but of sustained, organised campaigns.
The pattern principle also protects against a specific misuse risk: a community organised around the production of harm toward a protected group cannot use our tools to suppress that group by targeting individual pieces of content that appear ambiguous in isolation.
We assess harm at the pattern level as well as the content level.
Agency is the capacity of marginalised individuals to make choices, claim their dignity, and push back against systemic constraints.
Without it, targets of harm are reduced to passive victims; with it, agency becomes the primary tool for dismantling the architecture of inequality.
Targets of online harm currently have little to no agency in how to tackle it — they can report, block, or leave, and little else.
We want to change that. Whilst focusing initially on exposure control as our core value, we build to explore what collective agency looks like through peer support mechanisms and collective governance of models.
A model that protects one community by burdening another has not extended protection — it has relocated harm.
We build to extend safety, not to move it from one group to another. This runs through every annotation decision. When we label content as harmful, we ask who it harms and why — not only whether it feels harmful to us. A model that learns to suppress the speech of people with protected characteristics, even unintentionally, even because of how we labelled our training data, is not doing what we built it to do. We catch that by asking throughout the build — in schema design, in session facilitation, in labelling, in evaluation.
Some of the most important speech online is speech about harm, not speech that causes it: documentation, reporting, testimony, opposition to atrocity.
A model that cannot tell the difference between incitement to violence and documentation of violence will cause damage that’s hard to undo — silencing exactly the people it was built to protect.
We build our models to preserve that distinction.
We don’t believe that we are building the solution to online harm, because we don’t believe a tech solution can tackle oppression.
But we do believe it’s a vehicle for change.
And one that can contribute to our collective liberation from these systems.
Being able to encode the lived experience of online harm into datasets at this point in time feels incredibly important.
This is a time when regulation is falling into place (e.g., the Online Safety Act), platforms are being held accountable (Meta’s ruling), and the wave of resistance against Tech’s unaccountability to its users is being increasingly challenged by projects like DAIR, Ethical AI Alliance, etc.
- Benesch, S. (2018). Dangerous Speech: A Practical Guide.
- Citron, D. (2014). Hate Crimes in Cyberspace. Harvard University Press.
- Maronikolakis et al. (2022). Listening to Affected Communities to Define Extreme Speech. ACL.
- Rabat Plan of Action (2012). UN High Commissioner for Human Rights.
- Röttger et al. (2021). HateCheck: Functional Tests for Hate Speech Detection Models. ACL.
- Sap et al. (2020). Social Bias Frames. ACL.
- UK Equality Act 2010.
- UN Strategy and Plan of Action on Hate Speech (2019).
- Online Safety Act 2023.
- Waseem et al. (2017). Understanding Abuse: A Typology of Abusive Language Detection Subtasks. ACL.