# AI Quality Engineering Team Lead

**Company**: Aviva
**Location**: Warsaw, ASEC
**Work arrangement**: hybrid
**Experience**: senior
**Job type**: full-time
**Category**: Engineering
**Industry**: Insurance

**Apply**: https://aviva.wd1.myworkdayjobs.com/en-US/External/job/Poland---Warsaw---ASEC/AI-Quality-Engineering-Team-Lead_R-174666-1?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_24abdc38-91a

## Description

We are looking for an AI Quality Engineering Team Lead to help establish trustworthy, scalable quality engineering for GenAI and agentic AI solutions at Aviva.

You will combine hands-on evaluation with team leadership, helping stakeholders understand whether AI solutions are accurate, safe, reliable and ready for release.

**Responsibilities:**

- Leading the quality engineering approach for AI solutions, from risk assessment through release and monitoring.

- Evaluating multi-turn, RAG and agentic AI behaviour, including grounding, guardrails and tool use.

- Building representative datasets and assessing coverage, balance, leakage, label quality and freshness.

- Assessing correctness, relevance, hallucination, safety, bias, privacy and behavioural drift.

- Investigating unexpected outcomes across data, prompts, models, retrieval, integration, orchestration and evaluators.

- Building reusable evaluation components, automated test pipelines, reporting and evidence-generation patterns for multiple AI products.

- Combining test methods and validating automated evaluators against human judgement.

- Using baselines, key metrics and risk-based regression tests to support release decisions.

- Integrating evaluation into CI/CD, observability and engineering workflows with traceability across datasets, prompts, models and metrics.

- Producing concise, auditable findings that explain confidence, uncertainty, customer impact and residual risk.

- Partnering with product, engineering, architecture, data, risk, compliance and quality stakeholders to influence delivery and release decisions.

- Coaching and guiding AI Quality Engineers, planning team work, reviewing outputs and building a collaborative culture of constructive challenge and learning.

**Requirements:**

- Several years of relevant professional experience in quality engineering, software testing, software engineering, data science or a related technical field.

- Hands-on experience evaluating AI, large language model, RAG or agent-based applications.

- Practical Python skills and experience with APIs, logs, traces, structured datasets and SQL or similar analysis.

- Ability to design repeatable evaluations and automation, not just manual testing or vendor scoring.

- Good understanding of sampling, thresholds, trends and non-deterministic system behaviour.

- Experience with automated testing, version control, CI/CD and maintainable engineering practices.

- Ability to turn technical findings into decision-quality evidence for business, risk and engineering stakeholders.

- Experience coaching or leading engineers in an emerging or ambiguous domain.

- Strong communication and influencing skills, with the confidence to challenge weak evidence.

**Nice to have:**

- Experience leading projects, coordinating technical workstreams or managing people.

- Experience with AI evaluation or observability tools such as DeepEval, Arize or similar platforms.

- Experience designing shared frameworks, reusable services or platform capabilities.

- Knowledge of responsible AI, model risk, controls, privacy or customer-outcome requirements.

- Experience in financial services, insurance or another regulated industry.

- Exposure to cloud AI systems, observability and production support.

**Benefits:**

- Contract of employment

- Performance Bonus

- Private medical care (ENEL-MED)

- Cafeteria system of benefits (e.g. MultiSport card)

- Additional parent privileges: - baby leave – 6 weeks leave for second parent - shortened working time after parental leave

- Employer-funded group insurance

- Employee Shares Plan

- Extra day off for birthday

- Paid time off to volunteer

- Additional leave acquired over the years in the company (even 5 days more in a year)

- Wellbeing Leave - additional two weeks paid leave after 10 years of service

- Psychological support consultations

- Access to the e-learning platforms

- Internal employee referral program

- Hybrid working model from Poland with a few days in Warsaw office per month

- Attractive office location in Warsaw - Gdański Business Center

- Additional entitlements for people with disabilities - Work Without Barriers

- Work environment that promotes diversity - Diversity Charter

## Skills

### Required
- Python
- APIs
- logs
- traces
- structured datasets
- SQL
- automated testing
- version control
- CI/CD

### Nice to have
- DeepEval
- Arize
- cloud AI systems
- observability
- production support

---

Source: [Apply at aviva.wd1.myworkdayjobs.com](https://aviva.wd1.myworkdayjobs.com/en-US/External/job/Poland---Warsaw---ASEC/AI-Quality-Engineering-Team-Lead_R-174666-1?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)