# Lead Data Engineer

**Company**: Mastercard
**Location**: Pune, India
**Experience**: senior
**Job type**: full-time
**Category**: Engineering
**Industry**: Technology

**Apply**: https://mastercard.wd1.myworkdayjobs.com/en-US/CorporateCareers/job/Pune-India/Lead-Data-Engineer_R-289561?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_2f0b0d04-c2b

## Description

We are seeking a Lead Data Engineer to join our AI&DPE Data Engineering platform team. As a Lead Data Engineer, you will be responsible for building and evolving large-scale backend data systems, real-time and batch pipelines, and AI-enabled services that power analytics, decisioning, and automation across the organization.

**Primary Responsibilities**

- Design, develop, and maintain backend services and data pipelines using Scala, Python, and Java

- Build and optimize batch and streaming workloads on Cloudera Data Platform (CDP) using Spark

- Work with Cloudera Manager to support platform configuration, monitoring, performance tuning, and operational stability

- Design and implement high-quality datamarts and curated datasets with strong emphasis on data integrity, performance, and reliability

- Design, build, and integrate AI-powered agents that operate to support anomaly detection and operational intelligence workflow automation

- Apply generative AI-driven or rule-based agents to reduce manual effort and improve scalability across backend systems

- Build and support data and compute workloads in AWS environments

- Leverage Databricks for large-scale data processing and advanced analytics

- Contribute to cloud-native and hybrid architectures integrating on-prem and cloud platforms

- Enable downstream consumers through well-designed, reliable backend data interfaces

- Partner with analytics, fraud, and business teams to ensure backend systems meet evolving needs without compromising platform stability

- Create clear technical documentation, including architecture diagrams, data flows, and design specifications

- Participate in Agile/Scrum ceremonies and cross-functional design reviews

- Mentor and upskill team members in backend engineering, big data, and AI agent concepts

**Knowledge and Skill Requirements**

- BS/BA degree in Computer Science, Engineering, Information Systems, or related field

- Strong hands-on experience with Scala, Python, Java in backend or data-intensive systems

- Experience working with Cloudera Data Platform (CDP) and Spark

- Familiarity with Cloudera Manager for cluster administration, monitoring, or troubleshooting

- Strong understanding of data modeling concepts, distributed systems, and large-scale data processing

- Excellent problem-solving skills and ability to work independently in complex environments

## Skills

### Required
- Scala
- Python
- Java
- Cloudera Data Platform
- Spark
- Databricks
- AWS
- AI

### Nice to have
- Kafka
- Hadoop ecosystem
- Hive/Impala
- Informatica
- Agile/Scrum

---

Source: [Apply at mastercard.wd1.myworkdayjobs.com](https://mastercard.wd1.myworkdayjobs.com/en-US/CorporateCareers/job/Pune-India/Lead-Data-Engineer_R-289561?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)