Senior Data Engineer

The Department for Business, Innovation, Science and Trade (BIST) helps businesses invest, innovate, export and grow across the UK. The Digital, Data and Technology (DDaT) directorate helps to deliver this by designing, building and running the services, platforms and technology that businesses and colleagues rely on every day.

Data engineers at BIST are pivotal to our mission of driving economic growth by placing data at the heart of decision-making. We leverage leading cloud and open-source technology to build scalable, business-critical data pipelines, products and services, ensuring our users,whether within BIST, across government, or the public,have access to the right data in the optimal format.

Main responsibilities

• Lead the design, build, and testing of data products: Develop robust data products by integrating feeds from multiple systems, utilising a variety of storage technologies and access methods
• Ensure the delivery of resilient and scalable solutions: Select and deploy the most suitable technologies to create data solutions that are resilient, scalable, and future proof
• Have problem-solving expertise: Understanding of common issues in databases, data processes, products, and services, and apply standard solutions effectively
• Collaborate with stakeholders: Work closely with a diverse range of stakeholders to understand their requirements, establish project goals, and define deliverables.
• Support users and solve issues: Assist users in utilising data tools, resolve issues promptly, and escalate when necessary
• Promote best practices: Develop and advocate for best practices within the team.
• Encourage team building and development: Organise initiatives to foster team building and professional development
• Provide line management and support to junior team members, guiding their growth and development
• Drive continuous improvement: Stay informed about the latest trends and best practices in data engineering to drive continuous improvement

Person specification

• Expert Python and SQL skills: design and build reusable software, performance optimisation, automated testing, and troubleshooting complex production systems
• Experience leading teams to develop resilient Python data pipelines, including testing, automation, performance tuning and troubleshooting
• Proven ability to lead the design and delivery of secure, resilient AWS infrastructure, applying effective networking, access control, monitoring and secrets management through Infrastructure as Code using Terraform
• Experience building Docker and container-based workflows
• Broad experience developing and maintaining enterprise-level data solutions
• Experience defining and implementing data quality and automated testing strategies, including validation, monitoring and controls across complex data pipelines

Benefits

• Learning and development tailored to your role
• A flexible, hybrid working environment with options like condensed hours
• A culture encouraging inclusion and diversity
• A Civil Service pension with an average employer contribution of 28.97%
• Annual leave starting at 25 days rising to 30 days with service
• Three paid volunteering days a year
• An employee benefits programme including cycle to work