Stable Diffusion Trained On Explicit Child Images, Stanford Investigation Finds

Image generated on AI

Here’s a troubling downside to AI-generated artwork that probably never crossed your mind. According to an unsettling study by the Stanford Internet Observatory (SIO), popular AI text-to-image models like Stable Diffusion, created by Stability AI, were inadvertently trained on datasets containing illegal child sexual abuse material (CSAM). 

Stable Diffusion was trained using LAION-5B, a massive dataset comprising billions of images, which, as the research found, included hundreds of known CSAM images.

The Stanford researchers utilized PhotoDNA, a tool developed by Microsoft, to identify over 3,000 suspected CSAM pieces in the publicly accessible training data. This alarming number suggests that the actual amount of such content could be even higher. PhotoDNA works by matching the digital fingerprints of images to known CSAM in databases maintained by entities like the National Center for Missing and Exploited Children and the Canadian Centre for Child Protection.

This discomfiting finding raises serious questions about the ethical implications and safeguards in AI development, and the researchers are advocating for an immediate discontinuation of any models developed from Stable Diffusion 1.5 unless they incorporate stringent safeguards against such misuse.

While Stability AI’s Stable Diffusion is at the center of this report, other AI models like Midjourney were also trained on the same dataset. However, Google’s Imagen, fed with a different LAION dataset variant, LAION-400M, reportedly avoided using LAION datasets in subsequent iterations due to the discovery of inappropriate content, including pornographic imagery and harmful stereotypes.

Ben Brooks, head of public policy at Stability AI, has responded to the findings by reiterating the company’s commitment to preventing AI misuse, telling Forbes that Stability AI strictly prohibits using its models and services for unlawful activities, including editing or creating CSAM.

To mitigate these risks, the company has updated Stable Diffusion with new versions that filter out more “unsafe” and explicit material from training data and results.

This incident underscores the challenges in ensuring AI models are trained on datasets devoid of harmful content, particularly when dealing with extensive, publicly available datasets. The Stanford study calls for more rigorous measures in collecting and training datasets and in hosting models trained on scraped datasets, so as to minimize the presence of CSAM and other harmful content for safer and more responsible AI development.

“There are methods to minimize CSAM in datasets used to train AI models, but it is challenging to clean or stop the distribution of open datasets with no central authority that hosts the actual data,” the team acknowledges. “The report outlines safety recommendations for collecting datasets, training models and hosting models trained on scraped datasets. Images collected in future datasets should be checked against known lists of CSAM by using detection tools such as Microsoft’s PhotoDNA or partnering with child safety organizations.”

[via Forbes, Associated Press, Stanford Internet Observatory, cover image generated on AI]