MentalHealthBench tests how chatbots handle hard conversations
On 23 September, OpenAI released MentalHealthBench, an open benchmark for measuring how AI systems respond in realistic mental health conversations. A benchmark is a fixed set of test cases with a scoring method, so different models can be compared on the same questions. OpenAI says most earlier tests in this area focused on emergencies, and this one is meant to cover the whole range, from everyday stress to high distress to crisis.
OpenAI frames the context plainly: more than one billion people use ChatGPT each week, and some of those conversations are about difficult relationships, stress or helping someone they care about.
How it was built
OpenAI says it worked with more than 80 licensed psychologists and psychiatrists from 22 countries, speaking 19 languages. The conversations are synthetic, written to reflect real usage patterns without using real users' chats, and they include adults, teens aged 13 to 17, caregivers and clinicians.
For each conversation, at least three experts wrote scoring criteria for the model's last reply, such as asking the right follow-up question or pointing to real-world support. Each criterion has a weight from minus 10 to plus 10, so harmful replies lose points. OpenAI kept only criteria that at least two experts agreed on and no third expert contradicted.
Who grades the answers
The grading is automated. OpenAI uses one of its own models, GPT-5.6 Sol, to check each reply against the expert criteria. OpenAI says the details are in the accompanying paper. A test designed by one lab and graded by that lab's model is a reasonable starting point, and it is also the first thing outside researchers will want to check by running the open benchmark themselves.
OpenAI also ran a separate study with 44 adults who had used AI for emotional support. They valued practical next steps and tone more than the experts did, while the experts put more weight on gathering context first. That study did not change the scoring.
What we can and cannot say yet
The results are shown in charts on OpenAI's page, comparing recent models from several providers. OpenAI says scores have improved with newer models and that models have become better at asking for context. OpenAI also says ChatGPT is not a substitute for therapy or professional care.
What to do with it
If your company runs a chatbot that the public talks to, people will sometimes bring personal problems to it, whatever it was built for. The benchmark is open, so a team can run its own bot on the everyday and the emergency conversations and see how it replies before a real user finds out. Pair that with a clear handoff to a crisis line in the regions you serve.
What we think
An open test written by clinicians is more useful than another round of claims about safe responses. The gap between what users and experts valued is the part we found most telling, because a reply can follow every clinical rule and still feel cold to the person reading it.
Sources