Description
Thomson Reuters Labs builds and ships custom models for professional workflows requiring fiduciary-grade AI. The Thomson LLM family was developed by training leading open-weight models on decades of authoritative Thomson Reuters content, with thousands of hours of subject-matter-expert input.
As a research scientist on the evaluation team, you would be working with researchers to design, develop, optimize and scale the evaluation suite as Thomson scales to newer releases and product integrations. You’ll work in cutting-edge areas of AI development, such as multi-agent systems and decision models to keep the Thomson model suite at the frontier of specialized professional LLMs.
Key Responsibilities
- Build and operate the evaluation suite that runs Thomson evaluations. Leverage our existing suite to identify and understand the strengths and weaknesses of Thomson. Create new evaluations to meet the gaps in our coverage, based on a combination of product, external, and aspirational use cases for the model.
- Lead cross-functional teams in the creation of evaluation assets. Use your technical expertise to guide teams of scientists, engineers, and subject-matter experts working on evaluations and benchmarks. Mentor junior scientists in developing their own research and technical skills.
- Inform the technical direction of Thomson. Propose opportunities for improvements on topics such as more robust and reliable evaluation, new opportunities for applying LLMs, and optimizations on existing assets. Draw on evaluation results to inform our training objectives. Provide recommendations to senior leadership on technical trade-offs in model development.
- Conduct fundamental research. Develop new ideas in model evaluation across science of evaluations, new benchmarks, and safety and alignment. Publish research findings in top AI/ML/NLP conferences on a regular cadence. Integrate new research findings into model development and product pipelines.
Required Qualifications
- Hands-on research experience in evaluations – with examples of designing, building and releasing benchmarks for frontier models or agents, including research publications in top conferences such as NeurIPS/ICML/ICLR/*ACL.
- Experience working closely with researchers and cross-functional stakeholders to translate ideas into practical evaluations.
- Hands-on Python coding experience implementing benchmarks, agents or other systems built around LLMs.
- Familiarity with current developments in LLMs, with an interest in staying up to date with the evolving frontier.
- A PhD in Computer Science or a related field, or equivalent practical experience, plus industry experience working on post-training or evaluation.
Benefits
- Hybrid Work Model: We’ve adopted a flexible hybrid working environment for our office-based roles while delivering a seamless experience that is digitally and physically connected.
- Flexibility & Work-Life Balance: Flex My Way is a set of supportive workplace policies designed to help manage personal and professional responsibilities, whether caring for family, giving back to the community, or finding time to refresh and reset.
- Career Development and Growth: By fostering a culture of continuous learning and skill development, we prepare our talent to tackle tomorrow’s challenges and deliver real-world solutions.
- Industry Competitive Benefits: We offer comprehensive benefit plans to include flexible vacation, two company-wide Mental Health Days off, access to the Headspace app, retirement savings, tuition reimbursement, employee incentive programs, and resources for mental, physical, and financial wellbeing.
For more fresh new jobs, check out yubhub.co.