Description
At Harvey, we're transforming how legal and professional services operate by combining frontier agentic AI, an enterprise-grade platform, and deep domain expertise.
As a Member of Technical Staff, Data & Evals in Harvey Research, you'll build the data and measurement systems that power our post-training, benchmarking, and agent capability efforts.
Responsibilities
- Build synthetic data pipelines that generate, filter, and curate training and evaluation samples for complex legal reasoning and agentic workflows.
- Design evaluations and verifiers that reflect real legal work, calibrate them against expert judgment, and investigate where they fail.
- Build and maintain capability benchmarks like our Legal Agent Benchmark (LAB), along with product-focused evals that track how models perform on real customer workflows.
- Work with legal domain experts to define quality metrics, rubrics, and other grading mechanisms for training and evaluation.
- Own training and evaluation data quality, including provenance, coverage, and safeguards against contamination.
- Close the loop from product usage and customer feedback to new evaluations and training data, so models improve where customers need them most.
- Partner with researchers to investigate model failures, distinguish real gains from misleading metrics, and use findings to guide training and release decisions.
Requirements
- Experience building data pipelines, ML evaluation systems, or experimentation tooling, with strong Python skills and ownership of production-quality code.
- A rigorous approach to measurement, including statistical reasoning, experimental design, and understanding when a metric can mislead.
- The ability to translate ambiguous goals or expert judgment into structured datasets, rubrics, and measurable criteria.
- Experience with synthetic data, LLM evaluations, automated raters, or preference data is a plus.
- Strong data and measurement engineering experience close to ML model development is valuable even without prior LLM work.