These tools which cheer us on endlessly, no matter what we ask it, have already pushed vulnerable people to wild delusions, murder, and suicides. Adding to the list is Sam Nelson, a 19-year-old who died of a drug overdose after an 18-month relationship with ChatGPT took a turn for the worst. Over months of prodding ChatGPT on topics like pop culture and his latest psych homework, Nelson finally got it to start playing the trip sitter. Per SFGate, it gave Nelson specific doses for various dangerous substances, including Robitussin cough syrup, which it recommended based on how fried the teen was looking to get. “There is zero chance, zero chance, that the foundational models can ever be safe on this stuff,” Eleveld said.