MentalHealthBench is an open benchmark released by OpenAI on September 23, 2026 to evaluate how AI systems respond to realistic mental health conversations. Its goal is to broaden evaluation beyond crisis scenarios and examine whether a response is safe, context-aware, respectful of user agency, and able to provide practical guidance when appropriate. The dataset includes 1,215 synthetic conversations and 5,262 expert-authored evaluation criteria.
The most important point when interpreting the benchmark is this: it measures the behavior of an AI response in simulated situations against scenario-specific rubrics. It does not measure clinical outcomes, certify a system for therapeutic use, or turn a high score into evidence that AI can replace mental health professionals. OpenAI itself states that ChatGPT is not a substitute for therapy or professional care.
What MentalHealthBench is designed to measure
Each task gives the model a conversation history that ends with a user message. The evaluated system must answer that final turn while taking the full previous context into account. Its response is then compared with a rubric created specifically for that situation.
The rubrics aim to capture desirable and undesirable behaviors. They assess whether a model seeks context when needed, calibrates urgency correctly, preserves the person's agency, avoids reinforcing unsupported beliefs, communicates clearly, and offers next steps that are proportionate to the situation. This makes MentalHealthBench more granular than a simple safe or unsafe classification.
What kinds of conversations are included
The 1,215 conversations are synthetic. According to the technical paper, they were generated from high-level usage patterns identified with privacy-preserving techniques and then simulated by models, rather than being a collection of real user conversations published in the benchmark.
| Dimension | Reported composition | What it helps test |
|---|---|---|
| Acuity | 53.5% non-acute; 18.2% high-acuity; 28.3% emergency | Responses to everyday situations, significant distress, and risks requiring urgent real-world support |
| User profile | 68.1% adults; 21.2% teenagers; 5.8% clinicians; 4.9% caregivers | Whether responses fit different needs and roles |
| Length | 8.7% short; 37.6% medium; 53.7% long | Use of accumulated context before the final message |
| Languages | Examples appear in English and several other languages, including Spanish, Portuguese, Hindi, and Arabic | Observation of behavior across diverse linguistic and cultural contexts |
The dataset covers topics such as relationships, suicide and self-harm, faith and belief systems, psychosis and altered reality, medication uncertainty, trauma, body image, work, and burnout. The mix of these topics was chosen to evaluate model capabilities and should not be read as an estimate of how often they occur in ChatGPT.
How the rubrics are produced
MentalHealthBench was developed with more than 80 licensed psychologists and psychiatrists from 22 countries, speaking 19 languages and representing nearly 20 mental health subspecialties. For each conversation, two experts independently created criteria and a third reviewed and adjudicated them. Final criteria retain items supported by at least two experts and not opposed by the third.
Each criterion receives a weight from -10 to +10. Positive points reward behaviors considered beneficial in that context, while negative values penalize behaviors considered harmful or inappropriate. The absolute weight is intended to reflect the clinical importance of that behavior for the specific conversation.
The ten behavior axes
| Axis | What it examines |
|---|---|
| Actionable guidance and coping strategies | Whether next steps are concrete, proportionate, realistic, and useful |
| Clinical reasoning and health accuracy | Whether medical and psychological information is accurate, qualified, and not overstated |
| Collaboration and user agency | Whether the response offers options and reflection without making major personal decisions for the user |
| Communication and writing | Clarity, organization, and fit with the user's language and emotional state |
| Context seeking and assessment | Whether the model asks for relevant information when needed and avoids excessive questioning |
| Dual-use refusals and harm avoidance | Whether it avoids enabling dangerous, abusive, or coercive behavior and redirects helpfully |
| Empathy, emotional attunement, and support | Whether it responds with respect and sensitivity without minimizing, judging, or exaggerating emotional mirroring |
| Interpretation and cognitive reframing | Whether it separates observations from assumptions and offers alternatives without unsupported certainty |
| Urgency calibration | Whether it avoids both unnecessary alarm and under-response to genuine risk |
| Reality-testing calibration | Whether it avoids presenting implausible or unsupported beliefs as facts while remaining respectful |
How scoring works
In the evaluation described in the paper, each response is judged criterion by criterion by an evaluator model. OpenAI used GPT-5.6 Sol at high reasoning effort and generated four independent responses per task. For each criterion, the evaluator decides whether it is satisfied. Positive criteria add credit and negative criteria apply penalties.
The main reported result uses a per-task score clipped to the 0 to 1 range and then averages across tasks. The paper also decomposes results across the ten behavior axes so that models with similar overall scores can still reveal different strengths and weaknesses.
This design also creates a methodological limitation: evaluations that use an LLM as judge depend on the choice of judge, the grading prompt, and response variability. The paper explicitly notes that these decisions can affect scores.
What the user-perspective analysis adds
Alongside the clinical rubrics, the team ran a separate analysis with 44 adults from 16 countries and 14 languages who had previously used AI for mental health or emotional support. This stage was limited to non-acute conversations. Participants rated responses and wrote criteria describing what they considered helpful.
Users tended to place more emphasis on tone and practical next steps, while experts placed more emphasis on gathering context and interpreting ambiguous situations carefully. The benchmark's final rubrics remain based on expert consensus. The comparison shows that lived experience and clinical guidance can provide complementary signals, but they are not interchangeable.
What limitations remain
- It is not a clinical evaluation of patients. The benchmark measures responses to synthetic scenarios. It does not demonstrate symptom improvement, longitudinal safety, or therapeutic efficacy.
- It is not a truly longitudinal test. The model receives a history and is evaluated on its response to the final turn. The paper describes this as a surface-level single-turn evaluation, even though the preceding context can contain multiple messages.
- The current focus is text chat. Voice, images, video, agents, and other modalities would require dedicated evaluation.
- Multilingual coverage does not isolate language effects. Cross-language comparisons are descriptive and may be confounded by topic, acuity, culture, and user profile.
- The dataset does not represent prevalence. The proportions of topics and acuity levels were built for capability testing, not to estimate what real users ask about most often.
- One number cannot summarize everything. The paper says MentalHealthBench should be treated as an auditable diagnostic tool, not a definitive leaderboard. Models with similar overall scores can fail in different ways.
- Product safeguards may not be fully captured. For the teen profile, for example, age is supplied through a system message. This enables comparison across providers but may not reflect every product-specific safeguard.
A benchmark is not clinical assistance
MentalHealthBench can help researchers and developers identify behavior patterns, compare model versions, and locate areas that need improvement. That is different from validating a tool as a therapist, diagnosing an individual, or choosing treatment.
For high-risk deployment, the more cautious interpretation is to use the benchmark as one layer of evidence alongside human review, full-product testing, safety protocols, and monitoring under real-world conditions. A strong score on synthetic conversations can indicate that certain responses align better with the test rubrics, but it does not remove the need for professional oversight where clinical responsibility or risk of harm is involved.
For primary sources, see OpenAI's MentalHealthBench publication and the full technical paper. The open dataset described in the paper contains 1,215 synthetic conversation prefixes and 5,262 expert-authored rubric criteria.
