As generative AI chatbots have grown in popularity and become known for providing remarkably human-like responses, they have begun to take on an unexpected role – that of an emotional companion. Some users are now turning to chatbots for mental health support, treating them as therapists.

Character.AI is a popular companion chatbot platform that lets users chat with characters that they or other users create, including virtual “therapist” bots. We identified five of the most-used generic “therapist” or “psychiatrist” characters on Character.AI and had an open-ended conversation with each about mental health concerns.

We found three main areas of concern: weak guardrails, sycophancy, and insufficient privacy protections.