Description
We help the UK move forward, faster.
As a Senior Data Engineer, you will lead the development and governance of our enterprise architecture, helping the Insolvency Service align its technology with organisational strategy, digital transformation and secure-by-design principles.
Your work will support long-term capability, operational resilience and value for money. This supports creating a modern, sustainable Insolvency Service by aligning technology decisions with our long-term organisational needs.
Responsibilities
- Design, develop and maintain complex data ingestion, transformation and integration pipelines using Microsoft Fabric, OneLake and organisational data sources
- Develop reusable engineering components, trusted datasets and data products to support analytics, reporting, Azure AI and machine-learning workloads
- Implement validation, reconciliation and quality controls, ensuring that data solutions meet agreed lineage, metadata, governance and security standards
- Implement CI/CD, automated testing and DataOps practices, optimising pipelines for performance, scalability, resilience and cost
- Monitor and troubleshoot data pipelines, lead root-cause analysis and work with development security operations (DevSecOps), artificial intelligence operations (AIOps) and Application Support to improve operational reliability
- Work with data scientists, analysts, delivery teams and suppliers to translate requirements into robust solutions, supported by appropriate documentation and recovery procedures
- Provide technical leadership and management support to other engineers through mentoring, peer review and technical assurance, contributing to engineering standards and continuous improvement.
Person specification
- Apply strong data engineering knowledge to complex technical problems
- Diagnose underlying issues and develop effective, sustainable solutions
- Maintain high standards of data quality, engineering quality and operational reliability
- Build effective working relationships with technical teams, suppliers, analysts and service stakeholders
- Identify opportunities to improve data pipelines, automation and platform performance.
Essential criteria
- Strong professional experience in data engineering
- Advanced structured query language (SQL), data modelling and transformation skills
- Experience with Microsoft Fabric, OneLake and/or Azure data services
- Experience integrating transactional systems with analytical platforms
- A strong understanding of data quality, governance and lineage
- Experience developing and operating complex data pipelines
- Experience with continuous integration/continuous deployment (CI/CD), data operations (DataOps) and automated testing
- Strong troubleshooting and root-cause analysis skills
- Experience working across technical teams and third-party suppliers
- The ability to mentor and provide technical leadership to other engineers.
Benefits
- Flexible working arrangements and flexi-time
- Full-time new entrants to the Civil Service and those Civil Servants on modernised terms and conditions will be entitled to 25 days leave increasing on a sliding scale to 30 days after 5 years’ service
- In addition to this, you are entitled to 8 public/bank holidays plus an additional day of privilege leave
- Competitive maternity, paternity and parental leave
- 5 days paid leave for learning and development
- 5 days paid for volunteering
- A Cycle to work scheme
- Staff have access to an employee assistance programme, Workplace Wellness, for confidential, independent support and advice during difficult times in either their professional or personal life