Description
Mastercard is seeking a Senior Data Engineer to join its team in Hyderabad, India. This is an experienced individual-contributor role that owns end-to-end pipeline delivery and mentors peers.
The ideal candidate will bring cutting-edge software and full-stack development skills with advanced knowledge of cloud and data lake experience while working with massive data volumes.
Key Responsibilities:
- Own the end-to-end design, development, and delivery of complex batch and real-time data pipelines using Databricks, Spark, Python, and PySpark.
- Drive dbt-based transformation architecture with rigorous Test-Driven Development (TDD) and modular design.
- Design and own CI/CD pipelines using GitLab and Jenkins, and champion DataOps practices across the team.
- Administer and optimize Databricks objects and the platform using Databricks Asset Bundles, managing performance, access, and cost.
- Establish data observability frameworks covering quality, freshness, lineage, and anomaly detection.
- Lead the design of real-time data streaming pipelines (e.g., using Kafka, Spark Structured Streaming).
- Mentor junior and mid-level engineers, and lead code and design reviews.
- Partner with stakeholders on requirements, delivery planning, and technical trade-offs.
- Plan and execute deployments, migrations, and upgrades with minimal disruption to operations.
Required Qualifications:
- Bachelor's degree in computer science or a related technical field (Master's a plus).
- 5+ years of data engineering experience.
- Deep hands-on experience with Databricks (including administration), Spark, and Python.
- Strong background in PySpark and distributed data processing.
- Proven track record using dbt for robust, testable transformation workflows following TDD.
- Familiarity with Databricks Asset Bundles for object deployment and version control.
- Strong CI/CD (GitLab, Jenkins) and DataOps expertise.
- Experience with real-time data processing and streaming pipelines.
- Expert SQL development and strong cloud infrastructure experience (AWS, Azure, or GCP).
- Demonstrated experience mentoring engineers and leading technical delivery.
Ideally, you have experience in banking, e-commerce, credit cards, or payment processing and exposure to both SaaS and premises-based architectures.