Artificial intelligence in moderation: legal limits when blocking social media and gaming accounts

In the digital world of social media and online gaming, the use of artificial intelligence (AI) to moderate content and user behavior is becoming increasingly widespread. This technology enables platform operators to process large volumes of data efficiently and identify rule violations quickly. But what about the legal aspects, especially when it comes to blocking user accounts, which is done exclusively by AI systems and remains without human review?

Basics of AI moderation

AI systems in moderation analyze user behavior and content to detect violations of the terms of use. They are programmed to identify patterns that indicate inappropriate content, such as hate speech, harassment, copyright infringement or other forms of abuse. These systems can analyze not only text, but also images and videos, for example to detect pornographic or violent content. Their ability to process complex data patterns allows them to identify more subtle forms of rule violations that human moderators might miss.

In addition, AI systems can learn and adapt. Through machine learning and artificial neural networks, they are continuously improving their ability to identify relevant content and minimize false positives. This is particularly important as the way inappropriate content is presented is constantly changing and evolving.

The rapid response capability of this technology is a decisive advantage. Given the enormous amount of content that is generated daily on social media platforms and in online games, manually checking all content would not only be time-consuming, but also practically impossible. AI systems can work around the clock and react to potential violations in real time, enabling immediate and efficient moderation.

However, it is important to emphasize that these systems are not perfect. They can overlook or misinterpret context and nuances that are crucial to human understanding. Therefore, additional human review is essential to ensure the accuracy of moderation and to ensure that users’ rights are respected. Overall, AI systems provide valuable support in moderation, but require careful monitoring and regular adjustments to remain effective and fair.

Legal admissibility of AI-based blocks

The question of the legal admissibility of such automated blocking is complex and multi-layered. Decisions to block accounts must always be fair, transparent and comprehensible. A decision based solely on AI could fall short of these requirements, especially if there is a lack of human review. This could lead to unlawful blocking and thus potentially result in a claim for unblocking by the users or players concerned.

With regard to the General Terms and Conditions (GTC), it is essential that platform operators define clear guidelines and procedures for blocking accounts. These should explicitly mention the role of AI in this process and define the circumstances under which a block can be imposed. The GTC must also include the possibility of a human review and an objection procedure for users. Transparent communication of these guidelines is crucial to ensure legal compliance and maintain user trust.

The current case law, which prescribes a warning before a social media account is blocked, entails both risks for operators and opportunities for users. For operators, this means that they must carefully review their moderation processes and ensure that they comply with legal requirements. A premature or unjustified blocking without prior warning can have legal consequences and affect the credibility of the platform.

For users, however, this case law offers an additional layer of protection. It ensures that users are warned before their account is permanently blocked and have the opportunity to adjust their behavior or contest the decision. This can be particularly important in cases where the AI incorrectly classifies content or behavior as infringing. Users therefore have a greater chance of averting an unlawful block and unblocking their account.

Overall, the integration of AI into account moderation requires careful consideration of legal aspects. Compliance with current case law and the transparent design of general terms and conditions are crucial in order to both protect the rights of users and to protect operators from legal risks. This development underlines the need for a balanced and legally compliant approach to digital moderation.

The Network Enforcement Act (NetzDG) and AI moderation

The German Network Enforcement Act (NetzDG) plays an important role in the discussion about the use of AI in the moderation of online content. The law obliges social networks to remove or block obviously illegal content within a short period of time. In addition, the NetzDG requires transparency in the moderation processes, which means that platform operators must report on their content monitoring and removal procedures. This includes the disclosure of information on the use of AI systems and the frequency of their use.

Interestingly, however, the NetzDG does not go into detail when it comes to the specific way in which AI systems are implemented in the moderation process. Instead, it focuses on the effectiveness of measures to combat illegal content. This leaves some room for interpretation for providers as to how they integrate AI into their moderation processes, as long as the end results meet the requirements of the law.

Another important aspect of the NetzDG is the obligation to set up an effective complaints management system. Users must have the opportunity to object to decisions, including those made by AI. This underlines the need for platform operators to invest not only in advanced AI technologies, but also in robust review and complaints procedures to protect users’ rights and comply with legal requirements.

Overall, the NetzDG provides a framework that emphasizes the responsibility of social networks for the content on their platforms and at the same time highlights the importance of transparency and user rights. While the law does not provide specific guidance on the use of AI, it does set clear expectations regarding the outcomes and accountability of platforms.

Conclusion

The use of AI in the moderation of social media and online gaming raises important legal issues, particularly with regard to the blocking of user accounts. While AI systems offer efficiency and speed, they must comply with legal requirements for fairness and transparency. Exclusive reliance on AI decisions, without human review, could be legally problematic and lead to unblocking claims. The NetzDG provides a framework for moderation, but focuses more on the results than on the technologies used. In this dynamic legal environment, it is critical that social media and online gaming providers carefully consider both the technological and legal aspects of their moderation practices.

Author: Marian Härtel

Marian Härtel ist Rechtsanwalt und Fachanwalt für IT-Recht mit einer über 25-jährigen Erfahrung als Unternehmer und Berater in den Bereichen Games, E-Sport, Blockchain, SaaS und Künstliche Intelligenz. Seine Beratungsschwerpunkte umfassen neben dem IT-Recht insbesondere das Urheberrecht, Medienrecht sowie Wettbewerbsrecht. Er betreut schwerpunktmäßig Start-ups, Agenturen und Influencer, die er in strategischen Fragen, komplexen Vertragsangelegenheiten sowie bei Investitionsprojekten begleitet. Dabei zeichnet sich seine Beratung durch einen interdisziplinären Ansatz aus, der juristische Expertise und langjährige unternehmerische Erfahrung miteinander verbindet. Ziel seiner Tätigkeit ist stets, Mandanten praxisorientierte Lösungen anzubieten und rechtlich fundierte Unterstützung bei der Umsetzung innovativer Geschäftsmodelle zu gewährleisten.