# Senior Machine Learning Engineer, Agent Oversight

**Company**: Scale
**Location**: San Francisco, CA
**Experience**: senior
**Job type**: full-time
**Salary**: $216,000-$270,000 USD
**Category**: Engineering
**Industry**: Technology

**Apply**: https://job-boards.greenhouse.io/scaleai/jobs/4714527005?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_74465ecc-7b7

## Description

### About the Role

As a Senior Machine Learning Engineer on Agent Oversight, you will drive the end-to-end lifecycle ensuring production agents perform reliably and improve over time. This includes building observability tools, designing robust evaluation frameworks, and developing improvement loops.

### Responsibilities

- Build or contribute to observability into agent behavior in production , the signals and instrumentation needed to see what an agent is doing.

- Design evaluation methodologies and metrics for agentic applications, working with the platform to automate them at scale.

- Build, ship, and own ML systems detecting drift, anomalies, or misalignment in production agent behavior.

- Design and run rigorous experiments to validate model and agent performance improvements.

- Collaborate with software engineers, product managers, customers, and other teams to translate requirements into robust platform capabilities.

- Contribute to novel methods for agent evaluation and improvement or focus on building reliable ML systems at scale.

### Requirements

- 5+ years of experience as an ML engineer or applied scientist, ideally on a production ML or LLM-powered system.

- Strong grounding in at least two of: building/scaling evaluation/monitoring infrastructure, designing agent systems, or developing new methods.

- Hands-on experience with LLMs and agent architectures.

- Comfortable partnering with software engineers and collaborating across functions.

- Rigorous approach to experimentation and track record of giving/substantive feedback.

### Nice to Have

- Experience with RLHF, SFT, or other fine-tuning/RL workflows.

- Model or systems optimization experience.

- Published research, open-source contributions, or patents in agentic systems or applied ML.

- Experience working in regulated or enterprise contexts.

- Track record of taking a novel method from prototype to production.

### Compensation

The base salary range for this full-time position in San Francisco and New York is $216,000-$270,000 USD. You'll also receive benefits including comprehensive health, dental and vision coverage, retirement benefits, a learning and development stipend, and generous PTO.

## Skills

### Required
- Machine Learning
- LLMs
- Agent Architectures
- Evaluation Frameworks
- ML Systems

### Nice to have
- RLHF
- SFT
- Model Optimization
- Published Research
- Open-source Contributions

---

Source: [Apply at job-boards.greenhouse.io](https://job-boards.greenhouse.io/scaleai/jobs/4714527005?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)