AI gone weird

AI gone weird

Mark Zuckerberg thinks our BFFs will soon be AIs. He’s wrong.

By Brad Berens

The number of stories continues to increase about people surrendering all common sense when it comes to AI.

AI is God: In Rolling Stone ($), Miles Klee writes, “People Are Losing Loved Ones to AI-Fueled Spiritual Fantasies,” with the subtitle: “Self-styled prophets are claiming they have ‘awakened’ chatbots and accessed the secrets of the universe through ChatGPT.”

People fooling themselves into thinking that they have accessed the divine is nothing new. Nor is it new that people attribute consciousness to AI. As I’ve written before, 14-year-old Sewell Setzer III killed himself to be with a chatbot version of Daenerys Targaryen from Game of Thrones. What’s alarming about Klee’s piece is that GenAI enables prophetic self-delusion at scale.

One reason for this, as Mike Caulfield observes in “AI Is Not Your Friend” (The Atlantic, $) is that LLM GenAIs are often sycophantic:

This was not just a ChatGPT problem. Sycophancy is a common feature of chatbots: A 2023 paper by researchers from Anthropic found that it was a “general behavior of state-of-the-art AI assistants,” and that large language models sometimes sacrifice “truthfulness” to align with a user’s views. Many researchers see this phenomenon as a direct result of the “training” phase of these systems, where humans rate a model’s responses to fine-tune the program’s behavior. The bot sees that its evaluators react more favorably when their views are reinforced—and when they’re flattered by the program—and shapes its behavior accordingly.

So if some part of a user thinks, “God wants to talk with little old me,” then an AI chatbot will work to reinforce that delusion.

Instead of another example of self delusion, The New York Times ($) reports an example of deluding other people. The sister of a man killed in a road rage altercation used AI to resurrect a version of her brother to speak during the killer’s sentencing.

The sister found an image of her late brother, found audio of him speaking, wrote what she thought her brother would say, and worked with a technology company to synthesize a video of her dead brother making a family impact statement that a Maricopa County judge in Arizona—in an astonishing decision—allowed the family to play during sentencing. The defendant got the maximum penalty, and the judge said (on the record!), “’I loved that A.I.,’ Judge Lang said, describing the video’s message as genuine.”

Of course this happened in Maricopa County, home to former Sheriff Joe Arpaio whose hobby was humiliating inmates and immigrants. If there were ever grounds for appeal…

The new Pope Leo, in one of his earliest public statements, specifically mentions AI:

“In our own day, the church offers everyone the treasury of its social teaching in response to another industrial revolution and to developments in the field of artificial intelligence that pose new challenges for the defense of human dignity, justice and labor,” he said.

I don’t know what to make of this, but I’ll keep thinking. It’s interesting that the new pontiff did not mention AI as a threat to spirituality (ahem, see first AI story above).

Weirdest of all AI stories is Meta founder and CEO Mark Zuckerberg going on a PR blitz to talk about his vision for AI’s becoming our besties. In The Wall Street Journal ($), Meghan Bobrowsky has a good summary:

Mark Zuckerberg wants you to have AI friends, an AI therapist and AI business agents.

In Zuckerberg’s vision for a new digital future, artificial-intelligence friends outnumber human companions and chatbot experiences supplant therapists, ad agencies and coders. AI will play a central role in the human experience, the Facebook co-founder and CEO of Meta Platforms has said in a series of recent podcasts, interviews and public appearances.

“I think people are going to want a system that knows them well and that kind of understands them in the way that their feed algorithms do,” Zuckerberg said Tuesday during an onstage interview with Stripe co-founder and president John Collison at Stripe’s annual conference.

Zuckerberg said on a podcast last week that he thinks the average person wants to have more friends and connections with other people than they currently do—and that AI friends are a solution.

I hope you’ll excuse some bad language, but what the actual fuck?

Zuckerberg has created, purchased, or promulgated technologies that isolate users, train them not to talk with anybody who might disagree with them, make them ever more susceptible to misinformation and disinformation, and amplify loneliness so much that Vivek Murthy, Surgeon General in both the Obama and Biden administrations, called loneliness a national health crisis.

Now Zuckerberg’s answer to these problems is to make people even more isolated, talking only with sycophantic cyber buddies who will never disagree with you, tell you that you’re God, and coax you to buy things that you don’t need with money you don’t have?

Hearing about Zuckerberg’s latest demonstration that the man most interested in connecting people lacks the ability to connect with people himself reminded me of a passage from Douglas Adams’ The Hitchhiker’s Guide to the Galaxy (chapter 11):

The Encyclopedia Galactica defines a robot as a mechanical apparatus designed to do the work of a man. The marketing division of the Sirius Cybernetics Corporation defines a robot as “Your Plastic Pal Who’s Fun to Be With.”

The Hitchhiker’s Guide to the Galaxy defines the marketing division of the Sirius Cybernetics Corporation as “a bunch of mindless jerks who’ll be the first against the wall when the revolution comes,” with a footnote to the effect that the editors would welcome applications from anyone interested in taking over the post of robotics correspondent.

Curiously enough, an edition of the Encyclopedia Galactica that had the good fortune to fall through a time warp from a thousand years in the future defined the marketing division of the Sirius Cybernetics Corporation as “a bunch of mindless jerks who were the first against the wall when the revolution came.”

As always, Adams was prescient.

AI chatbots are not our friends. AI chatbots are a combination of math and language. You could argue that humans are similar constructs, but you’d be wrong.
__________

 

Brad Berens is the Center’s strategic advisor and a senior research fellow. He is principal at Big Digital Idea Consulting. You can learn more about Brad at www.bradberens.com, follow him on Blue Sky and/or LinkedIn, and subscribe to his weekly newsletter (only some of his columns are syndicated here).

 

 

Image Prompt (simple this time): “Create an image inspired by the text of this essay”

See all columns from the Center.

May 14, 2025