The police have seen a rise in AI-generated and deep-fake images in recent years – and say one of the biggest challenges is that it’s becoming “increasingly difficult to tell the difference between an AI-generated image and a genuine photograph”.
But Netsafe said many young people didn’t report online harm when they experienced it – meaning there could be incidents that never reach support services.
The threat is prompting some schools to change how they talk about online safety – and what they share online.
Christchurch Girls’ High School has been giving students and families advice about deep fakes, including warning about the risks of publicly sharing photos that can be manipulated using AI.
And Palmerston North’s Takaro School has gone a step further, telling parents it will stop posting photos of students on Facebook from next term. Photos from school events will instead be shared through its private school platforms.
Act MP Laura McClure has been pushing a member’s bill to criminalise the creation and distribution of non-consensual intimate deep fakes for the last 18 months.
It passed its first reading in Parliament earlier this year.
McClure said her bill built on existing laws around revenge porn, which make it illegal to share intimate images of someone without their consent.
She said an amendment to extend those protections to deep-fake and synthetic images was proposed by the Greens in 2022 – but it was not supported by Labour at the time.
“Look at where we are now. Our law isn’t up to date and we’re trying to play rapid catch-up – and the technology moves super fast.”
McClure said trying to regulate or ban the technology itself would be like playing “whack-a-mole”.
Instead, she said making the creation and distribution of non-consensual intimate deep fakes explicitly illegal would send a clear message to young people that there could be consequences.
She said existing revenge-porn laws had helped reduce harm among young people by making it clear the behaviour was illegal.
McClure said students are increasingly becoming aware of deep fakes and what AI can create.
“They know that it’s not technically illegal to do this right now. So there’s no real consequence necessarily for the actions.”
She said she had heard of increasingly disturbing cases, including a girls’ sports team whose team photograph was taken from a school newsletter and manipulated into a pornographic scene.
“I think it’s really horrible and really disgusting and really damaging.”
Netsafe chief online safety officer Sean Lyons said keeping pace with rapidly improving AI tools is becoming increasingly difficult.
“It can feel to many people like a bit of a game of whack-a-mole – we get one step ahead and then the bad guys get one step ahead.”
He said as the tools became better and easier to use, people would find new ways to misuse them.
Lyons said some young people also don’t report deep-fake incidents because of embarrassment and shame.
“Something that might be done as a ‘joke’, or part of banter, can also cause harm simultaneously, even if the intent was humour,” he said.
McClure said the consequences for victims could be severe – including being excluded from education, being unable to return to school or work or withdrawing from social and online spaces.
She said schools also didn’t get the same wraparound victim support they would if another crime happened on school grounds.
The ministry said it worked with Netsafe to support schools and young people dealing with online safety incidents, while providing guidance on generative AI and responsible digital technology use.
But McClure said deep fakes were adding another layer to the country’s existing bullying problem.
“We’ve got a huge problem in New Zealand with bullying in our schools … but there’s no oversight of how widespread the issue is.”
She said better data would help the education system target programmes – but for now, there is no clear picture of how this abuse is affecting our young people.
McClure said the issue was urgent, as the technology gets better.
Jaime Cunningham is a Christchurch-based reporter with a focus on education and general news. She joined Newstalk ZB in 2023 after working as a sports reporter at the Christchurch Star.
