# Research Engineer, Benchmarks & Evaluation

**Company**: Ramp
**Location**: New York, NY
**Work arrangement**: hybrid
**Experience**: senior
**Job type**: full-time
**Salary**: $204.4K - $352K
**Category**: Engineering
**Industry**: Finance

**Apply**: https://jobs.ashbyhq.com/ramp/d8925f84-562f-470c-a5bd-4f6c20009361?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_dca80872-c56

## Description

Ramp is building the smart infrastructure for finance teams, embedded in the transaction flow of every dollar a business spends.

You will own Ramp's benchmarking effort, turning real finance work into benchmarks and RL environments, building graders, and telling the team what is actually improving.

**Responsibilities**

- Build benchmarks from real workflows: turn agent traces, customer edge cases and expert labels into versioned benchmark suites for long-horizon agents that run for hours across tools and systems, plus tool use, multi-step reasoning and financial judgment

- Own the benchmarking effort: decide what we measure, run benchmark suites across models and agent versions, and report results the team makes decisions on

- Design graders: rubrics, programmatic checks and llm-as-judge, calibrated against human agreement, with a clear view of where each one breaks

- Close the loop with post-training: turn evals into reward functions and RL environments, and catch reward hacking and contamination before they ship

- Publish external benchmarks and write-ups on how models perform at real finance work

**Requirements**

- Hands-on experience building LLM benchmarks or evals that people used to make real decisions

- Strong Python and software engineering; your eval code is reproducible and fast

- Statistical judgment: variance, confidence intervals, sample size, inter-rater agreement, and knowing when a delta is noise

- Working knowledge of post-training (SFT, RL with verifiable rewards) and how evals feed into it

- Comfort with SQL and large datasets; you'll work directly from production traces

**Benefits**

- Flexible PTO

- Centralized home-office equipment ordering

- Health and wellness stipend

- Budget for intra-office travel

- Weekly coffee stipend

- 100% medical, dental & vision insurance coverage for you, with partial coverage for dependents (US)

- One Medical annual membership (US)

- 401(k), including employer match on contributions made while employed by Ramp (US)

- Fertility HRA (up to $10,000 per year) (US)

- Parental leave: up to 16 weeks (birthing + bonding) or 8 weeks (bonding only) at 100% pay (US)

## Skills

### Required
- Python
- software engineering
- LLM benchmarks
- eval code
- statistical judgment
- post-training
- SQL
- large datasets

### Nice to have
- industry experience on an eval, benchmarking or post-training team
- Built RL environments or verifiers for agentic tasks
- Published benchmark or evaluation research
- Built or contributed to eval frameworks
- Familiarity with accounting, finance or other expert-domain work

---

Source: [Apply at jobs.ashbyhq.com](https://jobs.ashbyhq.com/ramp/d8925f84-562f-470c-a5bd-4f6c20009361?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)