🔗 Share this article

Technology companies and child safety organizations will receive authority to assess whether artificial intelligence tools can produce child exploitation material under new UK legislation.

Significant Increase in AI-Generated Harmful Material

The announcement coincided with findings from a safety watchdog showing that cases of AI-generated child sexual abuse material have more than doubled in the last twelve months, growing from 199 in 2024 to 426 in 2025.

New Legal Structure

Under the amendments, the authorities will permit designated AI companies and child protection groups to inspect AI systems – the foundational systems for chatbots and image generators – and verify they have adequate safeguards to prevent them from producing images of child sexual abuse.

"Fundamentally about stopping exploitation before it happens," stated Kanishka Narayan, noting: "Specialists, under rigorous conditions, can now detect the risk in AI systems early."

Tackling Regulatory Challenges

The changes have been introduced because it is illegal to produce and own CSAM, meaning that AI creators and others cannot create such content as part of a evaluation process. Until now, authorities had to wait until AI-generated CSAM was uploaded online before dealing with it.

This law is designed to averting that problem by helping to stop the creation of those materials at their origin.

Legal Framework

The amendments are being added by the authorities as modifications to the crime and policing bill, which is also implementing a ban on possessing, producing or sharing AI models developed to create exploitative content.

Real-World Impact

This week, the minister toured the London headquarters of a children's helpline and listened to a mock-up conversation to counsellors involving a report of AI-based abuse. The call portrayed a teenager seeking help after facing extortion using a explicit AI-generated image of himself, created using AI.

"When I learn about children experiencing blackmail online, it is a source of intense anger in me and rightful concern amongst families," he stated.

Concerning Data

A leading online safety organization stated that cases of AI-generated exploitation material – such as online pages that may contain multiple images – had significantly increased so far this year.

Cases of the most severe content – the most serious form of abuse – rose from 2,621 visual files to 3,086.

• Girls were overwhelmingly targeted, making up 94% of prohibited AI images in 2025
• Depictions of infants to toddlers rose from five in 2024 to 92 in 2025

Industry Response

The legislative amendment could "represent a crucial step to ensure AI products are secure before they are launched," commented the head of the internet monitoring organization.

"AI tools have made it so victims can be targeted repeatedly with just a simple actions, providing offenders the ability to make potentially limitless amounts of advanced, lifelike exploitative content," she added. "Material which additionally commodifies victims' suffering, and renders young people, particularly girls, more vulnerable on and off line."

Counseling Interaction Information

Childline also released details of counselling sessions where AI has been mentioned. AI-related harms mentioned in the conversations comprise:

• Using AI to evaluate body size, physique and looks
• Chatbots discouraging young people from consulting trusted guardians about abuse
• Being bullied online with AI-generated material
• Online extortion using AI-manipulated images

Between April and September this year, Childline delivered 367 counselling sessions where AI, conversational AI and associated terms were discussed, four times as many as in the equivalent timeframe last year.

Fifty percent of the mentions of AI in the 2025 interactions were connected with mental health and wellbeing, including utilizing AI assistants for support and AI therapy apps.