# Member of Technical Staff - AI Evaluations, Health

**Company**: Microsoft AI
**Location**: Multiple Locations
**Work arrangement**: hybrid
**Experience**: senior
**Job type**: full-time
**Salary**: USD $119,800 – $234,700 per year or USD $160,200 – $261,000 per year for specific locations
**Category**: Engineering
**Industry**: Healthcare

**Apply**: https://microsoft.ai/job/member-of-technical-staff-ai-evaluations-health/?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_5f71138f-6aa

## Description

MAI Health is at the forefront of integrating AI into healthcare, partnering with the Mayo Clinic to deploy safe and effective AI agents. As a Member of Technical Staff - AI Evaluations, Health, you will play a critical role in ensuring these agents meet the highest standards of quality, factuality, and safety. The role involves working closely with clinicians and technical stakeholders to design and implement comprehensive evaluation strategies, metrics, and datasets that prioritize clinical accuracy and patient safety.

Responsibilities:

- Develop and own the end-to-end evaluation strategy for MAI Health's Mayo Clinic project, focusing on defining metrics, building datasets, and translating results into actionable steps for improving response quality.

- Create layered evaluations that cover a wide range of interactions, from static benchmarks to adaptive simulations and agentic evaluations.

- Collaborate with Mayo clinicians and subject-matter experts to design high-quality tooling and run evaluations for LLM-based tools, including writing clinically-grounded evaluation guidelines and recruiting physician raters.

- Design and write LLM-as-judge prompts for medical content, build automated evaluations, and report on judge-human agreement and inter-rater agreement.

- Evaluate patient-facing AI experiences for factuality, safety, and health literacy, ensuring outputs are trustworthy and safeguarded for vulnerable users.

- Serve as the technical partner for Mayo, gathering requirements, demonstrating capabilities, and representing partner needs to MAI engineering and research teams.

- Partner with data teams to develop scalable pipelines and dashboards for tracking evaluation performance.

- Contribute to privacy-preserving monitoring of production conversations and clinical review of concerning interactions.

- Translate evaluation insights into product improvements and inform launch decisions for clinical features.

- Monitor trends in model performance, clinician and patient feedback, and the clinical-AI benchmark landscape to refine evaluation approaches.

- Own key projects end-to-end, managing risks and proposing solutions to ensure timely delivery.

- Coordinate cross-team and cross-organization collaboration among MAI and Mayo stakeholders.

- Handle sensitive health data responsibly, adhering to HIPAA and partner data-governance requirements.

Qualifications:

- Bachelor's Degree in Computer Science or related technical field and 6+ years of technical engineering experience with coding in languages such as C, C++, C#, Java, JavaScript, or Python.

- 3+ years of experience leading multi-disciplinary projects, defining requirements, and executing with cross-functional teams.

Preferred Qualifications:

- Experience in healthcare, health tech, or regulated, safety-critical domains.

- Familiarity with clinical workflows, medical terminology, or working with clinician stakeholders.

- Background in computer science, experience in training/evaluation of LLMs, and technical depth in software development, data science, and machine learning.

## Skills

### Required
- C
- C++
- C#
- Java
- JavaScript
- Python
- LLM evaluation
- software development
- data science
- machine learning

### Nice to have
- healthcare
- health tech
- clinical workflows
- medical terminology
- HIPAA
- PHI handling
- FDA guidance on AI-enabled devices

---

Source: [Apply at microsoft.ai](https://microsoft.ai/job/member-of-technical-staff-ai-evaluations-health/?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)