AI Evaluation Engineer

As an AI Evaluation Engineer, you'll be an integral part of our AI Evaluation team, owning evaluation coverage for Dialpad's Agentic AI systems.

You'll design and execute validation strategies for agentic, NLP, and speech workflows across staging, beta, and release candidates.

Key responsibilities include:

• Building, running, and improving regression evaluations, A/B comparisons, and red teaming analyses
• Co-owning LLM-judge metric development, calibration, and prompt refinement
• Creating, configuring, and monitoring data annotation jobs
• Developing and maintaining QA tooling, notebooks, and pipeline components
• Investigating bugs, triaging issues, and deciding on escalations

Requirements:

• Bachelor's or Master's degree in Computer Science, Software Engineering, Computational Linguistics, or a related field
• 3+ years of experience in QA, test engineering, model evaluation, or applied ML quality for AI-driven products
• Experience designing structured test strategies and working with complex AI systems
• Strong analytical skills and experience collaborating with cross-functional teams

Benefits include competitive salary, comprehensive benefits, and real opportunities for growth.