Description
We are seeking an experienced Lead Data Engineer to join our billing transformation program. The engineer will be responsible for sourcing telemetry data, building scalable data pipelines, supporting rating logic, and preparing billing-ready outputs for SAP invoicing.
This is a critical role supporting a transformation from raw telemetry usage data to rated, validated, and invoice-ready billing data.
Key Responsibilities:
- Design and implement data ingestion pipelines from telemetry systems
- Develop curated usage datasets in Snowflake
- Implement rating logic and create billing-ready datasets
- Generate SAP invoice feeds
- Develop data quality and reconciliation frameworks
- Implement exception and error-handling processes
- Design CI/CD pipelines for data engineering deployments
- Develop monitoring, alerting, and operational dashboards
- Create technical documentation and runbooks
Required Skills:
- Strong hands-on experience in data engineering, ETL/ELT development, and large-scale data pipeline design
- Experience with AWS Glue, PySpark, Python, SQL, and cloud-based data processing
- Strong experience working with Snowflake as a data warehouse or enterprise data platform
- Good understanding of usage-based billing, rating engines, invoice feeds, or financial data processing
- Experience building data quality checks, reconciliation frameworks, audit controls, and exception handling processes
- Hands-on experience with CI/CD tools such as GitHub Actions, GitLab CI, Jenkins, AWS Code Pipeline, or similar
- Strong SQL skills for data analysis, validation, troubleshooting, and performance optimization
- Experience working with batch and event-driven data ingestion patterns
- Understanding of data modelling, partitioning, performance tuning, and cost optimization
Soft Skills:
- Strong analytical skills and attention to detail
- Ability to collaborate with cross-functional teams including Finance, Engineering, and Analytics
- Good communication skills to explain defects, logic, and test outcomes clearly
- High ownership and adaptability, with willingness to take on both QA and minor development tasks
Benefits:
- Hybrid working arrangement
- Annual Leave and Parental Leave benefits
- Upskilling opportunities with job-relevant training and certifications
- Dedicated Employee Assistance and Wellbeing support