Description
We are looking for an experienced Data Analyst who has built evaluation systems for complex technology products and is excited to take end-to-end ownership of quality in search built for AI agents.
In this position, you will:
- Own the quality measurement and improvement cycle end-to-end: identify user needs, define success, build evaluations, diagnose failures, and work with engineers and product managers to improve outcomes.
- Design, build, and evolve evaluation systems, including datasets, benchmarks, metrics, and automated pipelines that support development and launch decisions.
- Evaluate response quality across multiple dimensions, including relevance, completeness, freshness, factual accuracy, and usefulness to downstream agents.
- Investigate quality regressions and metric anomalies, uncover root causes, and identify the improvements with the greatest potential impact.
- Help prioritize the product and engineering roadmap, making quality trade-offs explicit and assessing whether changes deliver meaningful improvements for users.
We expect you to have:
- Substantial hands-on experience designing and building evaluation or quality measurement systems for complex technology products, such as search, AI agents, voice technologies, autonomous driving, recommendation systems, advertising, or other similar complex systems.
- A track record of owning ambiguous problems from initial framing through implementation, diagnosis, and measurable product improvement.
- Strong Python skills and an engineering mindset: you can turn analytical ideas into reliable, maintainable pipelines and tools.
- A solid foundation in mathematical statistics and practical experience with experimental design, sampling, uncertainty, and measurement bias.
- Strong analytical judgment: you understand what a metric captures, what it misses, and whether an observed improvement matters to users.
- The ability to reason across user needs, product behavior, and technical systems, and to translate that understanding into actionable evaluations.
- Strong communication and collaboration skills, with the ability to influence technical and product decisions.
- The independence to make progress in a fast-moving environment where requirements, methods, and definitions of success are still evolving.
Benefits & Perks:
- Competitive compensation
- Career growth and learning opportunities
- Flexibility and ownership
- Collaborative and innovative culture
- Opportunity to work on impactful AI projects
- International environment and talented teams