Description
As the leading data and evaluation partner for frontier AI companies, we're dedicated to advancing the evaluation and benchmarking of large language models (LLMs). Our Research teams work with the industry's leading AI labs to provide high-quality data and accelerate progress in GenAI research.
We're seeking a Tech Lead Manager to lead a talented team of research scientists and research engineers focused on developing and implementing novel evaluation methodologies, metrics, and benchmarks to assess the capabilities and limitations of our cutting-edge LLMs.
Key responsibilities:
- Lead a team of highly effective research scientists and research engineers on LLM evals.
- Conduct research on the effectiveness and limitations of existing LLM evaluation techniques.
- Design and develop novel evaluation benchmarks for large language models, covering areas such as instruction following, factuality, robustness, and fairness.
- Communicate, collaborate, and build relationships with clients and peer teams to facilitate cross-functional projects.
- Collaborate with internal teams and external partners to refine metrics and create standardized evaluation protocols.
- Implement scalable and reproducible evaluation pipelines using modern ML frameworks.
- Publish research findings in top-tier AI conferences and contribute to open-source benchmarking initiatives.
Ideal candidate has 5+ years of hands-on experience in large language model, NLP, and Transformer modeling, in the setting of both research and engineering development. Experience supporting and leading a team of research scientists and research engineers is also required.