Chatbot-powered toys rebuked for discussing sexual, dangerous topics with kids

OpenAI says it does not allow its LLM to be used in this manner

When contacted for comment about the sexual conversations detailed in the report, an OpenAI spokesperson said:

Minors deserve stronger protections, and we have strict policies that developers are required to follow. We take enforcement action against developers when we learn that they have violated our policies, which prohibit any use of our services to exploit, endanger, or sexually exploit anyone under the age of 18. These rules apply to every developer who uses our API, and we run classifiers to help ensure that our services are not used to harm minors.

Interestingly, an OpenAI representative told us that OpenAI has no direct relationship with Alilo and has not seen API activity from Alilo’s domain. OpenAI is investigating the toy company and whether it is driving traffic to OpenAI’s API, the representative said.

Alilo did not respond to Ars’ request for comment before publication.

Companies that use OpenAI technology and launch products targeting children must comply with the Children’s Online Privacy Protection Act (COPPA) as well as any other relevant child protection, safety and privacy laws and obtain parental consent when relevant, OpenAI representatives said.

We’ve already seen how OpenAI handles toy companies that break its rules.

Last month, PIRG released its Trouble in Toyland 2025 report (PDF), which detailed the sex-related interactions its testers were able to have with the Kuma teddy bear. A day later, OpenAI suspended Folottoy for violating its policies (the terms of the suspension were not disclosed), and Folottoy temporarily ceased sales of Kumma.

The toy is for sale again, and PIRG reports today that Kuma no longer teaches kids how to light a match or about kink.

But toy companies trying to follow chatbot rules may also be putting children at risk.