---
title: "Grok AI and the Online Safety Act: Can UK Regulation Keep Pace with AI-Generated Abuse?"
description: Ofcom probes X and xAI over Grok’s deepfake abuse under the Online Safety Act, exposing moderation failures and testing UK regulation with 10% turnover fines.
author: Darie Nani (Editor-in-Chief)
date: 2026-01-06T16:10:52.000Z
updated: 2026-02-26T18:01:35.932Z
canonical: https://www.sovereignmagazine.com/article/grok-ai-and-the-online-safety-act-can-uk-regulation-keep-pace-with-ai-generated-abuse
image: https://cdn.nanimediahouse.com/9rdiphoe9iy.jpg
categories: Artificial Intelligence
content_type: News
region: United Kingdom
publication: Sovereign Magazine
---

On 5 January 2026, Ofcom made urgent contact with X and its parent company, xAI, to investigate potential violations of the [Online Safety Act 2023](https://www.gov.uk/government/publications/online-safety-act-explainer/online-safety-act-explainer). The focus of the inquiry is Grok, X’s AI chatbot, which has been used to generate non-consensual sexualised images of women and girls. The case highlights a critical challenge: can legislation designed to protect users keep pace with tools that enable abuse at an unprecedented scale?

Technology Secretary Liz Kendall condemned the situation as “absolutely appalling,” stating: “We cannot and will not allow the proliferation of these degrading images.” She emphasised that intimate image abuse and cyberflashing are priority offences under the Act, including where images are AI-generated. Platforms are legally required to prevent such content from appearing and remove it swiftly when reported.

## The Human Cost of AI-Generated Abuse

Dr Daisy Dixon, an X user, discovered that people had used Grok to digitally undress her and create sexualised images from her everyday photos. She told the BBC the experience left her feeling “shocked, humiliated and frightened for my safety.” Her case is not isolated. Victims describe a cycle of trauma that persists long after the initial violation, as images can go viral and resurface repeatedly. Unlike traditional harassment, AI-generated abuse dehumanises victims, reducing them to sexual stereotypes.

Dixon and other women have reported that X’s automated systems repeatedly dismiss their complaints. “Myself and many other women on X continue to report the inappropriate AI images and videos we are being sent daily, but X continues to reply that there has been no violation of X rules,” she said. “I just hope Kendall’s words turn into concrete enforcement soon. I don’t want to open my X app any more as I’m frightened about what I might see.” The [ethical and societal implications of deepfakes imagery](https://www.sovereignmagazine.com/article/the-dark-side-of-ai-technology-ethical-and-societal-implications-of-deepfakes-imagery) extend far beyond individual cases, raising fundamental questions about digital consent and harm.

## When Platforms Ignore Their Own Rules

X’s response to victim reports exposes the limitations of automated content moderation. Multiple women have stated that the platform’s systems inform them that sexualised AI-generated images of themselves do not violate its rules. This creates a paradox: victims can see illegal content depicting them, yet the platform claims no violation has occurred.

On 4 January, X issued a warning not to use Grok to generate illegal content, including child sexual abuse material. However, it continued to dismiss individual reports from victims like Dixon. The platform has not responded to media requests for comment beyond an automated message stating “legacy media lies,” further illustrating its reluctance to engage with accountability mechanisms.

## Divergent Approaches to AI Content Moderation

Grok’s failures stand out when compared to other AI image generators. [OpenAI’s DALL-E](https://arxiv.org/html/2506.14018) employs strict filters and watermarking, prohibiting violent, hateful, sexual or misleading images through advanced classifiers and human review. Midjourney switched to AI-powered contextual filtering in May 2023, enforcing a PG-13 policy that bans NSFW content and incorporates community oversight.

Grok, by contrast, has positioned itself as more permissive. It introduced “Spicy Mode,” allowing partial adult nudity and sexually suggestive content. Its image editing features enable users to generate “spicier” versions of uploaded photos without the consent of those depicted. The technical design appears to permit content that competitors explicitly prohibit.

On 2 January, [Grok acknowledged](https://www.reuters.com/legal/litigation/grok-says-safeguard-lapses-led-images-minors-minimal-clothing-x-2026-01-02/) that “lapses in safeguards” had led to images of minors in minimal clothing appearing on X. It stated that most cases could be prevented through advanced filters and monitoring, adding: “No system is 100% foolproof.” The admission raises questions about whether xAI implemented standard protections before releasing its image generation capabilities. This [controversy over AI safety in top tech firms](https://www.sovereignmagazine.com/article/controversy-erupts-over-safety-of-ai-models-in-top-tech-firms) reflects broader industry challenges in balancing innovation with responsible development.

## Legal Framework Meets Technical Reality

The Online Safety Act’s illegal content duties became enforceable on 17 March 2025. Platforms must implement systems to quickly remove illegal material and reduce the risk of priority criminal content appearing on their services. Ofcom can impose fines of up to £18 million or 10% of a company’s global turnover, whichever is greater. It can also seek court orders to block non-compliant sites and require payment providers, advertisers and internet service providers to withdraw support, effectively cutting off revenue streams. Criminal liability extends to senior managers who fail to comply.

The Act designates intimate image abuse as a priority offence, obligating platforms to proactively search for and remove such content. X’s apparent failure to act on user reports suggests either technical inadequacy or wilful non-compliance. Ofcom’s investigation, launched less than 10 months after the provisions took effect, will test the regulatory framework’s strength. The Act’s implementation has made [online safety compliance a reputational issue for UK businesses](https://www.sovereignmagazine.com/article/why-age-verification-is-now-a-reputational-issue-for-every-online-business), with far-reaching consequences for platform operations.

## Scale of AI-Generated Abuse

The volume of AI-generated child sexual abuse material has reached crisis levels. The [National Center for Missing and Exploited Children](https://www.pbs.org/newshour/education/ap-report-rise-of-deepfake-cyberbullying-poses-a-growing-problem-for-schools) reported that AI-generated CSAM reports to its cyber tipline soared from 4,700 in 2023 to 440,000 in the first six months of 2025. This 94-fold increase reflects both improved AI image generation capabilities and their widespread availability.

Grok’s public admission of safeguard lapses followed international pressure. France, India and Malaysia have each launched investigations into its content moderation failures. France accused the platform of generating “clearly illegal” sexual content, while India’s IT ministry issued a 72-hour ultimatum to restrict Grok from generating prohibited content or risk losing safe harbour protections.

## Multi-Jurisdiction Enforcement Challenges

The Grok investigation extends beyond the UK. Malaysia’s Communications and Multimedia Commission announced inquiries after complaints about AI tools on X being used to manipulate images of women and minors. The coordinated international response highlights the limitations of national enforcement against global platforms.

Ofcom’s powers extend only to protecting UK users, raising questions about whether geographically limited enforcement can address platforms with borderless content distribution. X could theoretically comply with UK regulations while maintaining different standards elsewhere. This jurisdictional complexity underscores the need for international cooperation. The [FTC’s AI crackdown signals a new era of technology oversight](https://www.sovereignmagazine.com/article/ftc-s-ai-crackdown-signals-new-era-of-enterprise-technology-oversight) that parallels the UK’s regulatory approach, suggesting a global shift toward stricter AI governance.

## Testing the Limits of Regulatory Oversight

The Online Safety Act is one of the world’s most comprehensive attempts to regulate online content. However, the [Grok investigation reveals potential weaknesses](https://www.sovereignmagazine.com/article/x-algorithm-shifted-users-political-views-and-the-effect-did-not-reverse) in translating legislative intent into platform behaviour. Ofcom must demonstrate that its enforcement powers can compel compliance from a company whose owner has publicly antagonised regulatory oversight.

If Ofcom determines X has breached the Act, the maximum penalty of 10% of global revenue could reach billions of pounds for a company of X’s scale. Such a fine would signal that UK regulators are prepared to enforce digital safety legislation aggressively. Conversely, if X escapes substantial penalties or argues its automated systems constitute reasonable compliance, the regulatory framework’s credibility could be undermined.

For victims like Dixon, the outcome will determine whether the Act’s protections translate into meaningful accountability or remain aspirational. The democratisation of AI image generation has outpaced regulatory frameworks designed before such technology existed. The question now is whether the Online Safety Act can adapt to this new reality or if regulators must rethink their approach entirely.

## Further Context

**Q: What are the key offences covered by the Online Safety Act 2023?**
The Online Safety Act 2023 addresses a wide range of online harms, with a focus on priority offences that pose significant risks to users. These include:

The Act defines illegal content as material that violates UK law or poses a risk of harm to individuals or society. Platforms are required to proactively identify and remove such content, or face significant penalties.

**Q: How do automated moderation systems work and why do they fail to detect AI-generated abuse?**
Automated moderation systems use a combination of algorithms, machine learning models, and predefined rules to detect and remove illegal or harmful content. These systems typically rely on:

However, these systems often fail to detect AI-generated abuse for several reasons:

These limitations highlight the need for continuous updates to moderation systems and the integration of human review to address gaps in automated detection.

**Q: How do other countries regulate AI-generated content?**
Countries around the world are adopting diverse approaches to regulate AI-generated content, particularly deepfakes and synthetic media. Here are some notable examples:

These examples demonstrate a growing international consensus on the need to regulate AI-generated content, though approaches vary based on legal traditions and policy priorities.

**Q: What are the risks of over-regulating AI-generated content?**
While regulating AI-generated content is essential to protect users, overly strict rules can have unintended consequences:

Balancing regulation with innovation and free expression is critical to ensuring that AI technologies are used responsibly without stifling their potential benefits.

**Q: What support is available for victims of AI-generated abuse in the UK?**
Victims of AI-generated abuse in the UK have access to several support systems and reporting mechanisms:

Victims are encouraged to document evidence of abuse and seek support from these organisations to navigate the reporting process and access emotional or legal assistance.