AI Evaluation Engineer

By joining Sedgwick, you'll be part of something truly meaningful. It's what our 33,000 colleagues do every day for people around the world who are facing the unexpected. We invite you to grow your career with us, experience our caring culture, and enjoy work-life balance. Here, there's no limit to what you can achieve.

As an AI Evaluation Engineer, you will design, build, and operate the evaluation systems that determine whether our GenAI solutions are accurate, safe, reliable, and ready for production. You will serve multiple product pods, establishing consistent evaluation standards, automated pipelines, and quality gates, and you will help teams measure and improve LLM, RAG, and agentic applications throughout their lifecycle.

Responsibilities:

• Establish standardized, automated evaluation frameworks for LLMs and traditional ML models
• Deploy comprehensive observability solutions to monitor model drift, hallucination risk, latency, throughput, and token consumption
• Define evaluation methodologies and metrics for LLM, RAG, and agent-based applications
• Build reusable evaluation frameworks, templates, and libraries that pods can adopt with minimal effort
• Partner with product owners and SMEs to translate business requirements into measurable quality criteria and acceptance thresholds
• Develop automated evaluation pipelines integrated into CI/CD, with regression testing and quality gates for prompt, model, and retrieval changes
• Implement offline evaluation and online evaluation
• Support A/B and shadow testing for model, prompt, and architecture changes
• Design, validate, and maintain LLM-as-a-judge evaluators
• Establish human annotation workflows, labeling guidelines, and inter-annotator agreement practices with SMEs
• Track judge reliability over time and re-calibrate as models and use cases change
• Build and curate golden datasets, adversarial and edge-case suites, and synthetic test data generation approaches
• Apply approved data-handling practices for any dataset derived from sensitive or regulated data
• Implement red-teaming, hallucination, toxicity, PII leakage, prompt-injection, and jailbreak testing
• Produce evaluation evidence and reporting that supports security, privacy, compliance, and model risk reviews
• Contribute to responsible AI standards and release-readiness criteria
• Embed with pods to onboard them to evaluation practices, then coach teams toward ownership of their own eval suites
• Build dashboards and reporting that give engineering and business leaders transparent quality and cost/latency trends
• Evaluate and recommend eval tooling and platforms; document standards and best practices

Requirements:

• Master's degree in a computer science or a quantitative discipline from an accredited college or university preferred
• 4+ years in software, ML, or data engineering, with at least 1-2 years working hands-on with LLM-based applications
• Professional experience with evaluation and observability frameworks like LangSmith, Arize Phoenix, or WhyLabs
• Strong Python skills and software engineering fundamentals
• Practical experience evaluating LLM/RAG systems
• Hands-on experience with LLM-as-a-judge design, prompt engineering, and judge validation
• Working knowledge of statistics and experimental design
• Familiarity with at least one eval or observability tool
• Experience with cloud platforms and their managed LLM services
• Clear communication skills and the ability to work with engineers, product owners, SMEs, and risk stakeholders

Success Measures (first 12 months):

• Standard evaluation framework adopted by pods
• Automated quality gates in CI/CD for production-bound solutions
• Calibrated LLM judges with documented agreement against human labels
• Reduction in regressions reaching production and in time to validate model or prompt changes