Description
Ramp is building the smart infrastructure for finance teams, embedded in the transaction flow of every dollar a business spends.
You will own Ramp's benchmarking effort, turning real finance work into benchmarks and RL environments, building graders, and telling the team what is actually improving.
Responsibilities
- Build benchmarks from real workflows: turn agent traces, customer edge cases and expert labels into versioned benchmark suites for long-horizon agents that run for hours across tools and systems, plus tool use, multi-step reasoning and financial judgment
- Own the benchmarking effort: decide what we measure, run benchmark suites across models and agent versions, and report results the team makes decisions on
- Design graders: rubrics, programmatic checks and llm-as-judge, calibrated against human agreement, with a clear view of where each one breaks
- Close the loop with post-training: turn evals into reward functions and RL environments, and catch reward hacking and contamination before they ship
- Publish external benchmarks and write-ups on how models perform at real finance work
Requirements
- Hands-on experience building LLM benchmarks or evals that people used to make real decisions
- Strong Python and software engineering; your eval code is reproducible and fast
- Statistical judgment: variance, confidence intervals, sample size, inter-rater agreement, and knowing when a delta is noise
- Working knowledge of post-training (SFT, RL with verifiable rewards) and how evals feed into it
- Comfort with SQL and large datasets; you'll work directly from production traces
Benefits
- Flexible PTO
- Centralized home-office equipment ordering
- Health and wellness stipend
- Budget for intra-office travel
- Weekly coffee stipend
- 100% medical, dental & vision insurance coverage for you, with partial coverage for dependents (US)
- One Medical annual membership (US)
- 401(k), including employer match on contributions made while employed by Ramp (US)
- Fertility HRA (up to $10,000 per year) (US)
- Parental leave: up to 16 weeks (birthing + bonding) or 8 weeks (bonding only) at 100% pay (US)