Senior AI Evaluation Engineer (CI/CD & Platform)

We are seeking a Senior AI Evaluation Engineer (CI/CD & Platform) to own the continuous evaluation architecture, deployment pipeline infrastructure, and release gating for our enterprise AI Skills platform.

In this role, you will build both the MLOps/CI-CD deployment pipelines and the automated eval harnesses that ensure all production AI skills execute accurately, remain schema-compliant, and run safely in live customer environments.

Responsibilities

• Build tiered evaluation pipelines,separating instant static/schema checks for PR developer loops from comprehensive synthetic, multi-turn trajectory, and LLM-as-a-Judge eval suites for release promotion gates.
• Architect and maintain CI/CD build pipelines (GitHub Actions, GitLab CI), skill registry versioning, and runner environments to enforce automated "pass/fail" release gates on every pull request.
• Develop synthetic data generation pipelines and automated red-teaming utilities that allow skill authors to instantly generate edge-case evaluation suites from raw API schemas and domain specifications.
• Curate versioned golden datasets and execute continuous regression testing across base model upgrades, prompt changes, and security boundary checks.
• Ingest real-world production traces and failure logs back into benchmark datasets to continuously close test coverage gaps.

Requirements

• 4+ years of professional software engineering experience in Python building production-grade systems,with hands-on strength in EITHER CI/CD pipeline engineering OR AI evaluation methodology, plus working familiarity with the other.
• 2+ years designing, building, and operating test automation frameworks (pytest), CI/CD build pipelines (GitHub Actions, GitLab CI), or automated release gates.
• 1+ years of hands-on experience evaluating or building Generative AI applications (LLM tool-calling, agent workflows, or structured output parsing).
• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

Nice to Have

• AI Evaluation Tooling: Direct experience with modern evaluation and observability frameworks (e.g., LangSmith, Braintrust, DeepEval, Promptfoo, Phoenix).
• Schema Contracts & API Validation: Strong knowledge of schema validation libraries (JSON Schema, Pydantic, OpenAPI) and API contract testing.
• Agentic Standards & MCP: Familiarity with the Model Context Protocol (MCP), tool-calling specifications, or agent execution harnesses.
• Static Code Analysis: Experience using schema validation frameworks (Pydantic, JSON Schema), linters, and test automation tooling to enforce agreed code contracts.
• AI Security & Governance: Understanding of core LLM failure modes (hallucinations, prompt injection, PII/PHI sanitization, RBAC boundaries).
• Quality & Testing Discipline: Strong engineering rigor around test coverage, dataset versioning, evaluator calibration metrics, and regression prevention in continuous deployment pipelines.
• Technical Communication & Leadership: Demonstrated ability to own problem domains end-to-end, collaborate closely with platform and product teams, and write clear technical documentation and architecture designs.

Pay Information €96,000 EUR - €144,000 EUR