Description
Job Overview
As Lead Engineer, Data Platform, you will be a hands-on individual contributor leading platform initiatives within a clear technical direction set by the Data Infrastructure team. You will own small-medium platform initiatives and quarterly goals, taking them from design through to shipping and running them in production.
Responsibilities
- Design and deliver self-service paths so engineers can onboard new sources, datasets, and pipelines without a ticket queue, with access, masking, quality, and cost controls defined in infrastructure as code and CI/CD.
- Improve ingestion and transformation capabilities so data is fresher and more reliable, handling ordering, duplicates, late data, deletes, replay, and backfill, and migrate pipelines without breaking downstream tables.
- Apply governance in what you build: residency, PII masking, least-privilege and just-in-time access, retention and erasure, and lineage.
- Set SLOs, freshness and quality checks, and alerting that tells on-call what matters.
- Find the main cost drivers in data platforms and pipelines, and add guardrails and attribution that teams can act on.
- Break work into phases that ship, with clear milestones, a migration path, and a way to measure adoption.
- Use AI-assisted development to build and run the platform faster, and help prepare governed, well-described data that agents can use safely.
- Treat internal engineers as customers: write clear documentation and golden paths, and act on their feedback.
- Review designs and code for other engineers, and raise the bar for production-grade data and API work.
Requirements
- Independently designed, shipped, and operated a meaningful platform or data infrastructure initiative used by internal engineering teams.
- Solid data engineering fundamentals: batch and streaming ingestion, CDC and event-driven patterns, data modelling, and correctness.
- Strong production engineering in Python and SQL, including building and operating APIs, pipelines, or shared libraries.
- Practical infrastructure as code and CI/CD for data and infrastructure, including IAM across environments.
- Working knowledge of data governance in practice: PII masking, access control, residency, retention, and lineage.
- Observability for data pipelines: freshness and quality checks, SLOs, alert thresholds, and incident response.
- The ability to justify technology choices against requirements, with an order-of-magnitude view of cost and capacity.
- Hands-on use of AI-assisted development with sound engineering judgement.
- Clear written and verbal communication, and the ability to explain trade-offs to engineers and non-engineers across teams and time zones.
Nice to Have
- Experience with global, decentralised teams, and in financial services or other regulated environments.
Tech Stack
- GCP: BigQuery as our warehouse, Fivetran and event-driven ingestion, dbt, Apache Airflow, Looker, Atlan, Databricks, Terraform, GitHub, ConductorOne and Okta, Coralogix, and Claude for AI-assisted development.
- Platform tooling is written mainly in Python, and product teams build Java/Spring Boot services that publish events into the platform.
Benefits
- Competitive Compensation
- Time Off: Generous annual leave on top of bank holidays.
- Parental Leave: Paid maternity, paternity, and adoption leave.
- Sabbatical: Extended unpaid and paid leave options.
- Health Insurance: Private family insurance with additional OPD coverage and top-up options.
- Life & Accident Cover: Comprehensive accidental and life insurance protection.
- Mental Wellbeing: Access to therapy sessions, courses, meditations, and workshops.
- Volunteering & Development Days: Paid days annually for volunteering or personal growth.
- Learning & Development: Annual budget for books, courses, coaching, and more.
- WOO: Work from abroad for up to 90 days annually.
- Home Office Setup: Contribution towards setting up your home office.
- Laptop Ownership: Keep your old laptop and get a new one when it's time for a replacement.
- Snacks & Meals: Office perks with snacks, coffee, tea, and lunch.