Description
We are seeking an experienced and creative leader to own and continually improve product performance. This person will lead on how we define and measure safety, and partner across disciplines to strengthen our evaluations, safety systems, and user experience.
Key Responsibilities:
- Serve as the subject matter expert on safety systems, owning the safety roadmap and continuously improving guardrails as the product evolves.
- Develop a deep, data-driven understanding of product performance across safety risks; identify strengths, limitations, failure modes, emerging risks, and opportunities for improvement.
- Develop and evolve safety frameworks, benchmarks, evaluation methodologies, rubrics, quality standards, and meaningful safety metrics.
- Lead human and automated evaluation programs, including annotation programs, annotator and AI judge training and calibration, and improvements to evaluation reliability and scalability.
- Conduct qualitative and quantitative reviews from both expert and user perspectives, translating findings into clear, actionable product recommendations.
- Partner with Engineering, ML, Product, and Data Science to improve risk detection, prompting strategies, response generation, routing and escalation logic, automated evaluation pipelines, and the overall user experience.
- Prototype and test new safety approaches, product features, and user experiences to improve safety and broader product quality.
- Own ongoing safety monitoring and assessment, including dashboards, key metrics, trend analysis, emerging risk identification, safety reviews, and refinement of safety protocols.
- Evolve and own safety and launch-readiness criteria for new features and capabilities, identifying risks and ensuring appropriate mitigations are in place.
Qualifications:
- Advanced training (PhD, PsyD, LCSW, or equivalent experience) in psychology, behavioral science, mental health, or a closely related field.
- Deep expertise in psychological safety, crisis response, and risk assessment; strong judgment around conversations involving emotional distress, self-harm, suicide, abuse, or other sensitive situations.
- Experience developing evaluation frameworks or quality assurance processes.
- Excellent communication, analytical, and problem-solving skills with the ability to translate expertise into clear operational and product guidance for AI systems.
- Ability to collaborate effectively with engineers, researchers, designers, and product managers.
- Ability to make and communicate tradeoffs and clear decisions in ambiguous environments.
- Experience building with LLMs, conversational AI, or generative AI products.
- Up to date on current clinical AI best practices and emerging guidelines and standards.
Benefits:
- Healthcare coverage
- Annual incentive program
- Retirement benefits
Salary Range: $138,000 - $181,500