warnings are going to be given instead of bans
Discord: Predicting Harassment, Hate Speech, and Porn, and Horroring Content with AI Models
Improved warning systems and safety features are just some of the things that are coming to Discord in the coming weeks. “We’re making the [mobile] dark theme even darker and we’re adding a Midnight theme which is pure black,” explains Peter Sellis, Discord’s SVP of product. The wallpaper is hidden in the mobile apps and can be turned off to save battery life.
Most of the problematic posts on Discord aren’t nearly that grave, of course. As on any large platform, Discord fights daily battles against spam, harassment, hate speech, porn, and gore. It became a favored destination for fraudsters at the height of the coin craze.
It was argued that scanning isn’t invasion of privacy because it is using models instead of humans. In an interview with The Verge, John Redgrave, vice president of trust and safety, explained that the technology is a way to identify problematic content but it doesn’t feel like it is a violation of privacy. We are not going to invasion everyone’s privacy, while also providing tools that enrich people’s experience from a safety perspective.
Redgrave joined Discord two years ago after it acquired Sentropy, the company he co-founded to work on AI tools that detect harassment and abuse online. Discord is planning to expand these AI models beyond just blurring. “It gives us a mechanism by which we can introduce additional models in the future that can protect against other forms of challenging content,” explains Redgrave, who says Discord is also working on a grooming model to detect sexual exploitation on its service.
Discord: A Family Sharing Platform for Children, Families, and the Online Violence of a San Francisco Tech Company Exploring Online Violence and Murders
A similar opt-in feature for Apple’s Family Sharing setup was rolled out last year after privacy outcry forced Apple to drop some controversial child protection features. Apple can detect sexually explicit material in incoming and outgoing pictures and make them disappear if it spots it.
The search functionality on Discord mobile is also being improved soon with tappable search filters and an improved notifications tab with an auto-clear feature. Discord mobile is also getting a new Remix feature this week that lets you remix images into memes and share them on Discord.
If you’re interested in avatar decorations and profile effects, Discord’s in-app store is arriving for all users soon. It offers a bunch of decorations for profiles so your avatar can have an animation over it or people can preview your profile and see effects. Members of Nitro get a discount on the decorations and effects of their profile in the Discord Shop.
Last but not least, Discord is also rolling out some improvements for apps and developers. The premium app subscriptions that were previously US-only are extending to the UK and Europe this week. In addition, the main app is being tinkered with to make it usable in more places and to look into ways to allow users to use the app in a server with no need for an admin.
We should talk about the signs of a new reform movement in the traditional platform justice system. It hopes the initiative could lead to better behavior around the web if it succeeds at Discord.
Discord’s San Francisco campus is a tech company headquarters like many others, with its open-plan office, well stocked micro-kitchens and employees bustling in and out of over-booked conference rooms.
It’s clear from the glass doors at the entrance that this is a place built by players. On Wednesday, three employees were playing in a first-person shooter while sitting in a row at work.
Video games are designed for pure fun, but the community around those games can be notoriously toxic. In many dangerous cases, angry gamers call in the SWAT teams to their targets homes.
For Discord, which began as a tool for gamers to chat while playing together, gamers are both a key constituency and a petri dish for understanding the evolution of online harms. If it can hurt someone, there is probably an angry gamer somewhere trying it out.
A growing user base is one of the reasons why there is controversy over what users are doing on its server. The company made a lot of noise when leaked classified documents from the Pentagon were found on the platform. Discord faced previous scrutiny over its use in 2017 by white nationalists planning the “Unite the Right” rally in Charlottesville, VA, and later when the suspect in a racist mass shooting in Buffalo, NY was found to have uploaded racist screeds to the platform.
Inside Discord’s Reform Movement for Banned Users: Scaling Policy for Three-Strikes-and-You’re-Out
Some platforms have a three-strikes-and-you’re-out policy. Break the rules more than once. Your account will be nuked if you break them a third time. In many cases strikes are forgiven after 30 days, 90 days and so on. The nice thing about this policy from a tech company’s perspective is that it’s easy to communicate, and it “scales.” There is a way to build an automated system that issues strikes, reviews appeals, and does not require any human oversight.
It is not an appropriate policy to have three strikes. It levies the same penalty for both minor infractions and major violations. It doesn’t rehabilitate. Most users who receive strikes probably don’t deserve to be permanently banned, but if you want them to stay you have to figure out how to educate them.
Most platform systems don’t lack nuance. If a girl posts a picture depicting self-harassment, she will be taken down from the site. But the girl doesn’t need to be banned from social media — she needs to be pointed toward resources that can help her.
Details are one click away — From that message, users will be guided to a detailed modal that will give details of the post that broke our rules, outline actions taken and/or account restrictions, and more information regarding the specific Discord policy or Community Guideline that was violated.
However, some violations are more serious than others, and we’ll take appropriate action depending on the severity of the violation. For example, we have and will continue to have a zero-tolerance policy towards violent extremism and content that sexualizes children.
It acknowledges the need for social networks in the lives of people online, particularly young people, and it also acknowledges the idea that most users can rehabilitate if only someone would take the time to do so.
The new system has already been tested in a small group of servers and will begin rolling out in the coming weeks, Badalich said. Along with the new warning system, the company is introducing a feature called Teen Safety Assist that is enabled by default for younger users. When switched on, it scans incoming messages from strangers for inappropriate content and blurs potentially sensitive images in direct messages.
Source: Inside Discord’s reform movement for banned users
Why should Discord moderators be banned from a server? An open access team analysis of Discord’s reform reform for banned users
I appreciated the chance to sit in on the meeting, which was on the record, since the company is still in the early stages of building a solution. As in most subjects related to content moderation, untangling the various equities involved can be very difficult.
Alright, then. Perhaps moderators should be considered just as responsible for harms in a server as the owner? Well, it turns out that Discord doesn’t have a totally consistent definition of who counts as an active moderator. Some users are automatically given moderator permissions when they join a server. Why should they be held accountable if the server is rogue and the moderator isn’t posted in it?
It feels like an impossible knot to untangle. The team members found a way to analyse the server’s data, along with the behavior of the owners, as well as users, to find out why the server is malfunctioning.
Source: Inside Discord’s reform movement for banned users
What We Don’t Know About Trust and Safety: The Case of Over- and Under-Enforcing in a Multi-Degree System
It wasn’t perfect — nothing in trust and safety ever is. “The current system is a fascinating case of over- and under-enforcement,” one product policy specialist said, only half-joking. “What we’re proposing is a somewhat different case of over- and under-enforcement.”
I was quite confident that the company’s future systems would improve over time. Too often, trust and safety teams get caricatured as partisan scolds and censors. Visiting Discord gave a reminder that they can also be innovative.
