Description
We are looking for an AI Quality Engineering Team Lead to help establish trustworthy, scalable quality engineering for GenAI and agentic AI solutions at Aviva.
You will combine hands-on evaluation with team leadership, helping stakeholders understand whether AI solutions are accurate, safe, reliable and ready for release.
Responsibilities:
- Leading the quality engineering approach for AI solutions, from risk assessment through release and monitoring.
- Evaluating multi-turn, RAG and agentic AI behaviour, including grounding, guardrails and tool use.
- Building representative datasets and assessing coverage, balance, leakage, label quality and freshness.
- Assessing correctness, relevance, hallucination, safety, bias, privacy and behavioural drift.
- Investigating unexpected outcomes across data, prompts, models, retrieval, integration, orchestration and evaluators.
- Building reusable evaluation components, automated test pipelines, reporting and evidence-generation patterns for multiple AI products.
- Combining test methods and validating automated evaluators against human judgement.
- Using baselines, key metrics and risk-based regression tests to support release decisions.
- Integrating evaluation into CI/CD, observability and engineering workflows with traceability across datasets, prompts, models and metrics.
- Producing concise, auditable findings that explain confidence, uncertainty, customer impact and residual risk.
- Partnering with product, engineering, architecture, data, risk, compliance and quality stakeholders to influence delivery and release decisions.
- Coaching and guiding AI Quality Engineers, planning team work, reviewing outputs and building a collaborative culture of constructive challenge and learning.
Requirements:
- Several years of relevant professional experience in quality engineering, software testing, software engineering, data science or a related technical field.
- Hands-on experience evaluating AI, large language model, RAG or agent-based applications.
- Practical Python skills and experience with APIs, logs, traces, structured datasets and SQL or similar analysis.
- Ability to design repeatable evaluations and automation, not just manual testing or vendor scoring.
- Good understanding of sampling, thresholds, trends and non-deterministic system behaviour.
- Experience with automated testing, version control, CI/CD and maintainable engineering practices.
- Ability to turn technical findings into decision-quality evidence for business, risk and engineering stakeholders.
- Experience coaching or leading engineers in an emerging or ambiguous domain.
- Strong communication and influencing skills, with the confidence to challenge weak evidence.
Nice to have:
- Experience leading projects, coordinating technical workstreams or managing people.
- Experience with AI evaluation or observability tools such as DeepEval, Arize or similar platforms.
- Experience designing shared frameworks, reusable services or platform capabilities.
- Knowledge of responsible AI, model risk, controls, privacy or customer-outcome requirements.
- Experience in financial services, insurance or another regulated industry.
- Exposure to cloud AI systems, observability and production support.
Benefits:
- Contract of employment
- Performance Bonus
- Private medical care (ENEL-MED)
- Cafeteria system of benefits (e.g. MultiSport card)
- Additional parent privileges: - baby leave – 6 weeks leave for second parent - shortened working time after parental leave
- Employer-funded group insurance
- Employee Shares Plan
- Extra day off for birthday
- Paid time off to volunteer
- Additional leave acquired over the years in the company (even 5 days more in a year)
- Wellbeing Leave - additional two weeks paid leave after 10 years of service
- Psychological support consultations
- Access to the e-learning platforms
- Internal employee referral program
- Hybrid working model from Poland with a few days in Warsaw office per month
- Attractive office location in Warsaw - Gdański Business Center
- Additional entitlements for people with disabilities - Work Without Barriers
- Work environment that promotes diversity - Diversity Charter