Description
The Post Trade Division in LSEG has launched a Data Transformation program called Concorde. The objective is to have a new, centralised, and secure way to manage Post Trade data on cloud native technology in AWS and Snowflake.
As a Senior Data Engineer for the Concorde Platform Team, you will oversee Snowflake platform health, performance, and overall reliability, including troubleshooting ingestion, transformation, and orchestration issues.
Key responsibilities:
- Provide support to platform tenants on Snowflake and Apache Airflow, ensuring smooth operations across data pipelines.
- Develop and maintain an end-to-end testing framework to safeguard tenant flows during platform changes, while continuously optimizing queries and storage for performance and cost efficiency.
- Automate operational tasks through GitLab CI/CD pipelines and contribute to a consistent, well-managed release process.
- Document data architecture, pipeline processes, and best practices, and collaborate with the Snowflake team to evaluate and adopt new tools and frameworks.
- Actively support the release process to ensure consistent, high-quality platform delivery.
Requirements:
- Design and development of automated data pipelines and data structures
- 5+ years of working with Python Programming
- 3+ years hands-on data engineering experience for data warehouse solutions using Snowflake, SQL or similar
- Experience using Apache Airflow for workflow orchestration
- Hands-on experience with building CI/CD pipelines using GitLab or similar tools.
- Experience working with Agile Delivery teams (Scrum, Kanban) and JIRA
- Ability to build automated testing to test data pipelines
- Up-to-date knowledge of technology standards, industry trends, emerging technologies and software development best practices
Nice-to-have skills:
- Experience with managing infrastructure using terraform
- Experience with using dbt for data transformation and modelling
- Experience with kafka event streaming and messaging
Benefits:
- 25 days' holiday per year
- 1 additional day off for your Birthday
- Annual wellness allowance
- Share Purchase Plan
- Medical Health Insurance
- Medical Subscription (preventive medical services)
- Dental Coverage
- Life insurance
- Pillar 3 Private Pension
- Meal Vouchers
- Flexible Benefits
- Bookster
- Volunteering Events
- Team Buildings