Description
AI Platform builds the foundations of Datadog's AI efforts. The org is 70+ people organised in three pillars: training and serving, agents, and evaluation and experimentation.
This role sits in the evaluation and experimentation pillar, which owns Datadog's shared annotation and evaluation infrastructure , including the evaluation scenario store and the telemetry archival systems used across the Bits org.
Specifically, you'll be the first applied scientist on GenSim (Generative Simulations), the team that builds the environments Datadog's agents learn in.
What You'll Do:
- Own the applied science direction for GenSim: set the methodology and the forward-looking technical calls on how simulated environments and post-training data should be built.
- Define, measure and raise the quality of post-training data , basic correctness, representativeness against the real distribution of customer systems and production telemetry, and difficulty.
- Close the realism gap. Simulated environments today are too clean and the injected problems are not yet hard enough.
- Build scalable, production-grade systems rather than research scripts.
- Determine how this data is best applied, in LLM post-training and in evaluation.
- Work cross-functionally with the engineers and applied scientists on adjacent teams.
Who You Are:
- You have a PhD, MS or equivalent research experience in a scientific field, with strong applied mathematics grounding.
- 6+ years of relevant applied science or ML engineering experience, including setting technical direction for others.
- You have hands-on experience with LLM and agent post-training data.
- You have real domain expertise in LLMs and agentic applications.
- You have evaluated agents or LLM applications, and can define what 'good' means before you measure it.
- You are a strong programmer and production software engineer.
Bonus Points:
- Hands-on LLM fine-tuning, post-training or model training experience.
- Background in statistics, experiment design and data analysis.
- Experience deploying production-level ML infrastructure.
- Observability or monitoring systems background.
- Architecture-level understanding of LLMs.
Benefits & Growth:
- New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
- Continuous professional development, product training, and career pathing
- Intra-departmental mentor and buddy program for in-house networking
- An inclusive company culture and the ability to join our Community Guilds
- Access to Inclusion Talks, our internal panel discussions
- Free, global Spring Health benefits for employees and dependents age 6+
- Competitive global benefits