Research Engineer - Benchmarks

We are seeking a Research Engineer - Benchmarks to build and run independent, automated, public benchmarks of data providers and publish them weekly. The successful candidate will design and run head-to-head benchmarks, build and maintain test datasets, and own the automated harness and weekly benchmark release.

Responsibilities:

• Design and run head-to-head benchmarks of data providers against verified ground truth.
• Build and maintain test datasets and ground truth.
• Own the automated harness and the weekly benchmark release.
• Measure what buyers actually care about: accuracy, coverage, freshness, speed, and cost.
• Work with marketing to ship public leaderboard pages.
• Close the loop into Alexandria so benchmark results change which provider gets called for what.
• Keep expanding the categories we benchmark.

Requirements:

• 4+ years in ML, research engineering, or data engineering, with evaluation or benchmark work you've shipped.
• Experience with Python and API integrations.
• Strong understanding of building test sets and scoring methods.
• Ability to use AI heavily and keep upgrading your own workflow.
• Excellent communication skills.

Nice to Have:

• Experience with evaluation or benchmark work in a frontier lab, data company, or public benchmark project.
• Familiarity with messy vendor APIs, rate limits, and inconsistent schemas.

What We Offer:

• Salary: $250,000–$290,000 USD/year (San Francisco), $210,000–$224,000 CAD/year (Toronto).
• Competitive equity.
• Generous PTO: 15 days mandatory, anything after 24 days, just ask.
• Parental leave: 12 weeks fully paid.
• Wellness stipend: $100 USD/month.
• Learning & Development: Expense up to $1,000 USD/year.
• Team offsites.
• Sabbatical: 3 paid months off after 4 years.

Benefits for US-based full-time employees:

• Full coverage medical, dental, and vision.
• Life & Disability insurance.
• Virtual care and a health guide.
• Mental health: Talkspace.
• Fertility and family building: Carrot.
• EAP: Free confidential counseling.
• 401(k) plan.

Benefits for Canada-based full-time employees:

• Full coverage extended health, dental, and vision.
• Life & Disability insurance.
• Virtual care: Dialogue Premium.
• Mental health: Talkspace Elite.
• Fertility and family building: Carrot.
• Retirement: Group RRSP.