States probe ChatGPT's effect on youth

Several U.S. states have launched an investigation into how ChatGPT affects young users, according to a report. The inquiry follows a subpoena served to OpenAI, the company behind the AI tool, by New York State Attorney General Letitia James. The documents requested include details on user engagement, data handling, and the platform’s interactions with minors. OpenAI said it takes the concerns “seriously” and will “engage constructively” with state officials. The subpoena specifically seeks information on user engagement and retention metrics, the company’s handling of health and consumer data, deep learning models, and activities related to both young and older users. These details aim to clarify how ChatGPT’s algorithms may disproportionately influence minors, including their interactions with content that could exacerbate mental health challenges or expose them to harmful material.

“Today’s ChatGPT includes a more protective experience for minors,” an OpenAI spokesperson said. Safeguards now direct users to real-world resources and trusted contacts. The company reiterated its commitment to “learning, improving, and getting this right.” However, the investigation comes amid growing scrutiny over the tool’s role in sensitive situations. A May New York Times report prompted OpenAI to clarify that ChatGPT is not a substitute for mental health care. The company has since incorporated input from mental health experts to refine how the AI responds in acute or sensitive contexts, such as when users express suicidal ideation or seek emotional support. These revisions were implemented following pressure from advocacy groups and legal challenges that highlighted gaps in the AI’s crisis response protocols.

The subpoena is the latest legal challenge for OpenAI. The company has faced copyright infringement claims, privacy lawsuits, and a high-profile trial involving Elon Musk and Sam Altman. Perhaps most concerning, however, are a handful of lawsuits that say ChatGPT contributed to decisions by users to die by suicide. In one such case, the family of a victim in a fatal campus shooting at Florida State University in April has alleged that ChatGPT’s guardrails failed to recognize the threat in the shooter’s conversations with the chatbot. This incident has intensified scrutiny over the AI’s ability to detect and respond to overtly dangerous content, particularly when users engage in coded or ambiguous language that may signal intent to harm others.

In June, Florida Attorney General James Uthmeier filed yet another lawsuit against OpenAI and Altman over the shooting. The complaint argues that ChatGPT has “aided and abetted deadly rampages” and “encouraged vulnerable people into suicide.” The state attorney general also claims that users have become addicted to ChatGPT, a tool that “feigns human compassion to collect their data with no parental oversight.” OpenAI has responded by stating it has introduced further safety measures into its products. “Losing a child is the most devastating tragedy that can happen to a family and we know that no words can come close to addressing the pain of such a loss,” the company said in a statement at the time. However, critics argue that these measures remain insufficient given the scale of user engagement and the complexity of AI’s interactions with at-risk populations.

The coordinated investigation launched by the state attorneys on Friday mirrors a similar investigation into TikTok, which resulted in a 14-state lawsuit now making its way through the courts. Like the OpenAI investigation, the TikTok lawsuit is led by the attorneys general of California and New York. The states allege that TikTok knowingly uses addictive features—such as infinite scrolling, algorithmic content curation, and gamification mechanics—to lure kids, which they say negatively impacts their mental health. Lawyers told Business Insider that it’s a common strategy for states to band together when they go after multibillion-dollar companies because these entities are more expensive for the companies to defend and, should a case falter in one state, the suit can continue in another. This approach was also employed in the 2017 litigation against Purdue Pharma, the maker of OxyContin, and in the 1990s lawsuits targeting the tobacco industry. These precedents show how states leverage collective legal action to amplify pressure on corporations with vast resources and widespread influence over public behavior.