Senior Machine Learning Engineer, Agent Oversight

About the Role

As a Senior Machine Learning Engineer on Agent Oversight, you will drive the end-to-end lifecycle ensuring production agents perform reliably and improve over time. This includes building observability tools, designing robust evaluation frameworks, and developing improvement loops.

Responsibilities

• Build or contribute to observability into agent behavior in production , the signals and instrumentation needed to see what an agent is doing.
• Design evaluation methodologies and metrics for agentic applications, working with the platform to automate them at scale.
• Build, ship, and own ML systems detecting drift, anomalies, or misalignment in production agent behavior.
• Design and run rigorous experiments to validate model and agent performance improvements.
• Collaborate with software engineers, product managers, customers, and other teams to translate requirements into robust platform capabilities.
• Contribute to novel methods for agent evaluation and improvement or focus on building reliable ML systems at scale.

Requirements

• 5+ years of experience as an ML engineer or applied scientist, ideally on a production ML or LLM-powered system.
• Strong grounding in at least two of: building/scaling evaluation/monitoring infrastructure, designing agent systems, or developing new methods.
• Hands-on experience with LLMs and agent architectures.
• Comfortable partnering with software engineers and collaborating across functions.
• Rigorous approach to experimentation and track record of giving/substantive feedback.

Nice to Have

• Experience with RLHF, SFT, or other fine-tuning/RL workflows.
• Model or systems optimization experience.
• Published research, open-source contributions, or patents in agentic systems or applied ML.
• Experience working in regulated or enterprise contexts.
• Track record of taking a novel method from prototype to production.

Compensation

The base salary range for this full-time position in San Francisco and New York is $216,000-$270,000 USD. You'll also receive benefits including comprehensive health, dental and vision coverage, retirement benefits, a learning and development stipend, and generous PTO.