Description
Howden Group Services is expanding its internal data engineering capability and is looking for a highly experienced Senior Data Engineer with deep expertise in Databricks and Azure data services.
The ideal candidate will have a strong background in designing and delivering scalable, metadata-driven data solutions, particularly in metadata ingestion and orchestration processes.
The role requires a strategic thinker who can lead technical design discussions, drive best practices, and mentor junior engineers.
Responsibilities
The successful candidate will:
- Lead the development of metadata ingestion frameworks and orchestration processes.
- Design and implement scalable and reusable metadata-driven services to optimise data pipelines.
- Architect and maintain robust data solutions using Databricks and Azure services (such as Autoloader, Spark Declarative Pipelines Azure Data Factory (ADF), Azure Functions, and Azure Data Lake Storage (ADLS)).
- Establish and enforce best practices for data engineering, CI/CD pipelines, and DevOps for data.
- Collaborate with business stakeholders, data scientists, and analysts to understand data needs and translate them into efficient engineering solutions.
- Optimise and enhance existing data pipelines for performance, reliability, and scalability.
- Drive the adoption of data governance and security best practices.
- Lead the design and maintenance of ETL processes.
- Mentor and support junior engineers, sharing best practices and driving a culture of continuous improvement.
Requirements
Candidates should have:
- 8+ years of experience in data engineering with a strong focus on Databricks (SQL & PySpark).
- Extensive experience with Azure data services, including ADF, ADLS, and Azure Functions.
- Proven track record of designing and implementing metadata-driven data pipelines.
- Deep expertise in orchestration and data workflow automation (e.g., Airflow, DBT).
- Strong understanding of CI/CD practices for data engineering.
- Experience with infrastructure as code (Terraform, Bicep, or ARM templates) (preferred).
- Solid development and coding standards, with experience in software engineering best practices.
- Experience leading technical design discussions and driving architectural decisions.
- Strong stakeholder management skills, with the ability to translate business requirements into data solutions.
- Knowledge of data governance, security, and compliance best practices.
- Experience working with insurance data (not essential, but preferred).
What do we offer in return?
A career that you define. At Howden, we value diversity – there is no one Howden type. Instead, we're looking for individuals who share the same values as us:
- Our successes have all come from someone brave enough to try something new.
- We support each other in the small everyday moments and the bigger challenges.
- We are determined to make a positive difference at work and beyond.