# Research Engineer - Benchmarks

**Company**: Firecrawl
**Location**: San Francisco, CA
**Work arrangement**: hybrid
**Experience**: senior
**Job type**: Full time
**Salary**: $250K - $290K
**Category**: Engineering
**Industry**: Technology

**Apply**: https://jobs.ashbyhq.com/firecrawl/90f88e13-7d5f-43e2-a3fc-9e16d836b71c?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_6d2c87e7-ac7

## Description

We are seeking a Research Engineer - Benchmarks to build and run independent, automated, public benchmarks of data providers and publish them weekly. The successful candidate will design and run head-to-head benchmarks, build and maintain test datasets, and own the automated harness and weekly benchmark release.

Responsibilities:

- Design and run head-to-head benchmarks of data providers against verified ground truth.

- Build and maintain test datasets and ground truth.

- Own the automated harness and the weekly benchmark release.

- Measure what buyers actually care about: accuracy, coverage, freshness, speed, and cost.

- Work with marketing to ship public leaderboard pages.

- Close the loop into Alexandria so benchmark results change which provider gets called for what.

- Keep expanding the categories we benchmark.

Requirements:

- 4+ years in ML, research engineering, or data engineering, with evaluation or benchmark work you've shipped.

- Experience with Python and API integrations.

- Strong understanding of building test sets and scoring methods.

- Ability to use AI heavily and keep upgrading your own workflow.

- Excellent communication skills.

Nice to Have:

- Experience with evaluation or benchmark work in a frontier lab, data company, or public benchmark project.

- Familiarity with messy vendor APIs, rate limits, and inconsistent schemas.

What We Offer:

- Salary: $250,000–$290,000 USD/year (San Francisco), $210,000–$224,000 CAD/year (Toronto).

- Competitive equity.

- Generous PTO: 15 days mandatory, anything after 24 days, just ask.

- Parental leave: 12 weeks fully paid.

- Wellness stipend: $100 USD/month.

- Learning & Development: Expense up to $1,000 USD/year.

- Team offsites.

- Sabbatical: 3 paid months off after 4 years.

Benefits for US-based full-time employees:

- Full coverage medical, dental, and vision.

- Life & Disability insurance.

- Virtual care and a health guide.

- Mental health: Talkspace.

- Fertility and family building: Carrot.

- EAP: Free confidential counseling.

- 401(k) plan.

Benefits for Canada-based full-time employees:

- Full coverage extended health, dental, and vision.

- Life & Disability insurance.

- Virtual care: Dialogue Premium.

- Mental health: Talkspace Elite.

- Fertility and family building: Carrot.

- Retirement: Group RRSP.

## Skills

### Required
- Python
- API integrations
- ML
- data engineering
- evaluation
- benchmarking

### Nice to have
- frontier lab
- data company
- public benchmark project
- messy vendor APIs

---

Source: [Apply at jobs.ashbyhq.com](https://jobs.ashbyhq.com/firecrawl/90f88e13-7d5f-43e2-a3fc-9e16d836b71c?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)