# Senior AI Evaluation Engineer (CI/CD & Platform)

**Company**: Workday
**Location**: Dublin
**Work arrangement**: hybrid
**Experience**: senior
**Job type**: full-time
**Salary**: €96,000 EUR - €144,000 EUR
**Category**: Engineering
**Industry**: Technology

**Apply**: https://workday.wd5.myworkdayjobs.com/en-US/Workday/job/Ireland-Dublin/Senior-AI-Evaluation-Engineer--CICD---Platform-_JR-0110918?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_c664a1d2-c4c

## Description

We are seeking a Senior AI Evaluation Engineer (CI/CD & Platform) to own the continuous evaluation architecture, deployment pipeline infrastructure, and release gating for our enterprise AI Skills platform.

In this role, you will build both the MLOps/CI-CD deployment pipelines and the automated eval harnesses that ensure all production AI skills execute accurately, remain schema-compliant, and run safely in live customer environments.

**Responsibilities**

- Build tiered evaluation pipelines,separating instant static/schema checks for PR developer loops from comprehensive synthetic, multi-turn trajectory, and LLM-as-a-Judge eval suites for release promotion gates.

- Architect and maintain CI/CD build pipelines (GitHub Actions, GitLab CI), skill registry versioning, and runner environments to enforce automated "pass/fail" release gates on every pull request.

- Develop synthetic data generation pipelines and automated red-teaming utilities that allow skill authors to instantly generate edge-case evaluation suites from raw API schemas and domain specifications.

- Curate versioned golden datasets and execute continuous regression testing across base model upgrades, prompt changes, and security boundary checks.

- Ingest real-world production traces and failure logs back into benchmark datasets to continuously close test coverage gaps.

**Requirements**

- 4+ years of professional software engineering experience in Python building production-grade systems,with hands-on strength in EITHER CI/CD pipeline engineering OR AI evaluation methodology, plus working familiarity with the other.

- 2+ years designing, building, and operating test automation frameworks (pytest), CI/CD build pipelines (GitHub Actions, GitLab CI), or automated release gates.

- 1+ years of hands-on experience evaluating or building Generative AI applications (LLM tool-calling, agent workflows, or structured output parsing).

- Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

**Nice to Have**

- AI Evaluation Tooling: Direct experience with modern evaluation and observability frameworks (e.g., LangSmith, Braintrust, DeepEval, Promptfoo, Phoenix).

- Schema Contracts & API Validation: Strong knowledge of schema validation libraries (JSON Schema, Pydantic, OpenAPI) and API contract testing.

- Agentic Standards & MCP: Familiarity with the Model Context Protocol (MCP), tool-calling specifications, or agent execution harnesses.

- Static Code Analysis: Experience using schema validation frameworks (Pydantic, JSON Schema), linters, and test automation tooling to enforce agreed code contracts.

- AI Security & Governance: Understanding of core LLM failure modes (hallucinations, prompt injection, PII/PHI sanitization, RBAC boundaries).

- Quality & Testing Discipline: Strong engineering rigor around test coverage, dataset versioning, evaluator calibration metrics, and regression prevention in continuous deployment pipelines.

- Technical Communication & Leadership: Demonstrated ability to own problem domains end-to-end, collaborate closely with platform and product teams, and write clear technical documentation and architecture designs.

**Pay Information** €96,000 EUR - €144,000 EUR

## Skills

### Required
- Python
- CI/CD pipeline engineering
- AI evaluation methodology
- Generative AI applications
- schema validation libraries

### Nice to have
- LangSmith
- Braintrust
- DeepEval
- Promptfoo
- Phoenix
- JSON Schema
- Pydantic
- OpenAPI
- Model Context Protocol (MCP)

---

Source: [Apply at workday.wd5.myworkdayjobs.com](https://workday.wd5.myworkdayjobs.com/en-US/Workday/job/Ireland-Dublin/Senior-AI-Evaluation-Engineer--CICD---Platform-_JR-0110918?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)