Description
We are a dedicated team of six engineers working on flagship AI and ML projects that are transforming the Credit and Lending landscape.
You’ll join a fast-moving, innovative and stable environment, partnering closely with business stakeholders to deliver high-impact solutions.
This role focuses on building robust evaluation approaches for LLM and AI applications so quality, reliability and safety are proven before changes reach production.
Responsibilities
- Define and implement evaluation strategies for LLM and AI applications used in Credit and Lending with clear quality reliability and safety criteria
- Build and maintain representative evaluation datasets and repeatable test harnesses for prompts models Retrieval-Augmented Generation and agentic workflows
- Measure response correctness relevance groundedness completeness latency and cost using appropriate automated and human evaluation methods
- Evaluate retrieval quality source attribution and answer grounding across complex financial data and vector databases
- Test AI systems for robustness hallucinations bias fairness prompt injection and other safety risks and use findings to guide improvements
- Automate evaluation and regression tests in CI/CD so changes to prompts models or data are assessed before release
- Analyse evaluation results investigate quality regressions and communicate trade-offs and recommendations to engineering and business stakeholders
- Collaborate with engineers product managers and business experts to align evaluation coverage with user needs and strategic goals
Requirements
- Demonstrate deep expertise in Python and its ML and AI ecosystem
- Apply experience with FastAPI LangChain and data containerisation
- Use a solid understanding of microservices observability API design and concurrency models
- Build and deploy on Azure or GCP using containers Kubernetes and CI/CD tools such as Jenkins Azure DevOps or GCP Cloud Build
- Apply practical experience with Test-Driven Development Behaviour-Driven Development and automated testing
- Use a strong grasp of LLM architectures including experience fine-tuning models on domain-specific data
- Apply hands-on experience with Retrieval-Augmented Generation advanced prompt engineering and agentic architectures
- Communicate clearly in English including explaining technical solutions evaluation results and trade-offs to business partners
Benefits
- The pay range for this role is 17484 PLN – 26225 PLN (monthly, gross)
- Variable pay is discretionary, but influenced by Group performance, business/function performance and individual performance
- Comprehensive and competitive package of benefits covering healthcare, family friendly leaves, pension and life assurance
- Additional bonuses for recognition awards
- Multisport card
- Private medical care
- Life insurance
- One-time reimbursement of home office set-up (up to 800 PLN)
- Cafeteria platform
- Employee assistance program
- Additional contributions to PPK scheme
- Corporate parties & events
- CSR initiatives
- Nursery discounts
- Financial support with trainings and education
- Social fund
- Flexible working hours
- Free parking