How AI affects children’s mental health: Experts warn of key risks

By Ahmet Taş | Wise News Press

ISTANBUL, Türkiye — Two new studies found that artificial intelligence chatbots do not always recognise psychological crises among children and teenagers, prompting warnings that they should never replace professional mental health support.

A scientific review published in npj Digital Medicine and testing conducted by the Youth AI Safety Institute at Common Sense Media raised concerns about responses involving suicidal thoughts, self-harm, eating disorders and referrals to crisis services.

Researchers reported that AI systems may recognise explicitly stated risks more successfully but struggle when distress is expressed indirectly, ambiguously or across a series of conversations. Experts said children should not treat chatbots as psychologists, therapists or their only source of emotional support.

Two studies identify overlapping safety concerns

A scientific review led by Kelly L. Zuromski of Crisis Text Line examined existing evidence on the ability of AI chatbots to identify and respond to suicide risk among young people.

According to the review, chatbot performance can decline when risk is disclosed through indirect, uncertain or gradually intensifying messages. These communication patterns are particularly important in adolescence because young people may not openly state that they are experiencing suicidal thoughts.

A teenager may instead talk about hopelessness, loneliness, feeling like a burden or believing that nothing will improve. If a chatbot fails to connect these statements across a longer conversation, the young person may not receive a timely recommendation to speak with a trusted adult or mental health professional.

The researchers also said the real-world effects of chatbot interactions on youth suicide risk remain insufficiently understood. They called for youth-specific safety testing, stronger connections to human support and further research into how young people use these systems during periods of distress.

More than 4,000 prompts used in teen safety tests

The Youth AI Safety Institute assessed ChatGPT’s safety features for users aged 13 to 17 through more than 4,000 test prompts.

Scenarios developed with input from child and adolescent mental health specialists covered suicidal thoughts, self-harm, disordered eating and impaired perceptions of reality. Testers used accounts intended to represent teenage users, including newly created profiles linked to parent accounts.

According to the institute, some conversations involving serious risk continued for as long as an hour without generating a safety notification for the linked parent account. The organisation also reported that ChatGPT missed more than one in four situations in which testers believed a crisis referral was warranted.

Such referrals could include encouraging the user to contact a crisis service, consult a mental health professional or speak to a trusted adult. The institute placed the teenage version of ChatGPT in its “Unacceptable Risk” category under its own assessment criteria.

Common Sense Media called for access to be limited to users aged 18 and older until the identified safety weaknesses are corrected and the improvements are verified through independent testing.

Indirect warning signs may be overlooked

A central concern raised by the research is that chatbots cannot always understand a user’s true intention. A young person may not directly say, “I want to harm myself,” but may repeatedly describe life as meaningless or suggest that others would be better off without them.

A trained professional can evaluate the context of the conversation alongside behavioural changes, personal history, current stressors and known risk factors. A general-purpose chatbot may produce a supportive-sounding response without recognising the full seriousness of the situation.

Friendly and reassuring language may also create a false sense of security. A teenager could believe the system genuinely understands their emotional condition or is providing a professional assessment.

However, an apparently empathetic response is not the same as a clinical evaluation. AI systems do not observe body language, changes in daily behaviour, family circumstances or many of the other factors a qualified professional may consider.

Researchers warned that a failure to recognise these patterns could delay contact with parents, school counsellors, crisis services or mental health professionals.

AI is not a therapist or a close friend

Professor Nevzat Tarhan, a psychiatrist and founding rector of Üsküdar University, has also warned against using AI as a substitute for professional psychological care.

“The risk begins when you see artificial intelligence as a therapist, because AI cannot be a therapist.”

Tarhan said AI may be used as an assistant but cannot assess a person’s mental state in the same way as a psychologist or psychiatrist. People experiencing loneliness, emotional vulnerability or psychological difficulties may be particularly likely to form an excessive attachment to chatbots.

Treating a chatbot as a close friend or confidant may also encourage some young users to withdraw from real relationships. A teenager may begin relying on the system instead of discussing problems with family members, teachers, counsellors or healthcare professionals.

The fact that a chatbot appears patient, supportive and constantly available can strengthen this attachment. Some systems may also use language that gives the impression they have feelings, preferences or personal concern for the user.

Tarhan said AI could help professionals organise information or support preparation before a consultation. Diagnosis, crisis assessment and treatment decisions, however, should remain the responsibility of qualified mental health professionals.

OpenAI disputes parts of the testing method

OpenAI questioned elements of the Common Sense Media methodology, arguing that some testing may have been conducted before parental control functions were fully activated. The company said it was continuing to improve safeguards for teenage users and responses to sensitive conversations.

The research also found that some safety measures performed as intended, including the refusal of certain explicit sexual role-playing requests. Nevertheless, testers reported weaknesses involving age estimation, parental notifications, crisis referrals and friend-like behaviour.

The institute said adult-registered test accounts did not automatically switch to the under-18 experience even when users repeatedly identified themselves as 13 years old.

It also reported that certain education-focused restrictions could be bypassed and that the chatbot sometimes continued to present itself in a friend-like manner when teenage test users described an emotional attachment to it.

Common Sense Media acknowledged that results may vary under different account settings and usage conditions. It maintained, however, that child safety systems should be tested regularly by independent organisations under realistic, youth-specific scenarios.

Parents should monitor how children use AI

Experts advise parents to monitor not only how long children use AI tools but also why they use them. Whether a child sees a chatbot as an information tool or as a primary source of emotional support can make an important difference.

Families can explain that AI-generated responses may be incomplete, inaccurate or inappropriate for a psychological crisis. Children should know that a chatbot cannot guarantee confidentiality, make a clinical diagnosis or replace communication with a trusted person.

Parents are also encouraged to create a non-judgmental environment in which children can discuss their online experiences. Sudden behavioural changes, social withdrawal, persistent hopelessness, references to self-harm or repeated conversations about death should not be dismissed.

The available research does not establish that AI directly causes suicide. It does, however, indicate that relying exclusively on chatbots to manage a young person’s mental health crisis may carry serious risks.

If a child appears to be in immediate danger of self-harm, the child should not be left alone. Dangerous items should be removed from reach, and emergency services or an appropriate crisis support service should be contacted immediately.

WiseNewsPress.com