# Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data Architecture, Apache Airflow) **Company**: Mastercard **Location**: Pune, India **Experience**: senior **Job type**: full-time **Category**: Engineering **Industry**: Finance **Apply**: https://mastercard.wd1.myworkdayjobs.com/en-US/CorporateCareers/job/Pune-India/Principal-Data-Engineer--AWS--Databricks--Ai--ML-Flow--Data-Architecture--Apache-Airflow-_R-291404?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply **Canonical**: https://yubhub.co/job/principal-data-engineer-aws-databricks-ai-ml-flow-c3e521f5-d2e/ ## Description Mastercard Foundry R&D develops emerging technology solutions and transforms promising concepts into scalable products and platforms. We are seeking a Principal Data Engineer to design and build the data foundations that power advanced analytics, machine learning, generative AI, and agentic AI solutions. **Responsibilities** - Lead the architecture, design, and engineering of scalable batch, streaming, and event-driven data platforms. - Design and build reliable data ingestion, transformation, enrichment, processing, and consumption pipelines. - Develop governed lakehouse and modern data platform architectures using cloud-native technologies and distributed data processing frameworks. - Build and operate platforms that support analytics, machine learning, generative AI, and agentic AI workloads. - Create reusable data products, services, APIs, and self-service capabilities that accelerate experimentation and production delivery. - Enable the full AI and machine learning lifecycle through robust feature engineering, training, evaluation, deployment, and inference pipelines. - Optimize data workloads for performance, scalability, reliability, resiliency, and cost efficiency. - Define engineering standards for data modeling, software development, testing, CI/CD, observability, and operational excellence. - Implement data governance capabilities including quality controls, lineage, metadata management, cataloging, access control, retention, and compliance. - Apply security and privacy-by-design principles for sensitive and regulated data environments. - Evaluate emerging data and AI technologies through prototypes, proof-of-concepts, and technical assessments. - Translate loosely defined research, innovation, or product requirements into practical architectures and incremental delivery plans. - Provide hands-on technical leadership through architecture reviews, code reviews, troubleshooting, and engineering mentorship. - Partner with cross-functional teams to mature successful R&D initiatives into enterprise-grade production solutions. - Communicate technical decisions, trade-offs, risks, dependencies, and roadmap recommendations to both technical and business stakeholders. - Drive adoption of modern data engineering practices, platform automation, and platform reliability disciplines. **Requirements** - Typically 12-18 years of overall career relevant experience, including significant ownership of complex enterprise data engineering solutions. - Extensive experience designing, building, and operating large-scale production data platforms. - Expert programming skills in Python, PySpark, and modern software engineering practices. - Advanced SQL expertise, including data modeling, performance tuning, query optimization, and large-scale analytical processing. - Strong hands-on experience with distributed data processing technologies such as Apache Spark and modern lakehouse platforms. - Experience building and operating cloud-native data platforms in AWS, or Azure, or other enterprise cloud environments. - Strong knowledge of modern data architecture patterns, including data lakes, lakehouses, data meshes, data products, and event-driven architectures. - Experience with cloud storage, data integration, streaming, serverless, observability, and security services across public cloud platforms. - Experience building scalable batch and real-time data pipelines. - Experience with workflow orchestration platforms such as Apache Airflow, Databricks Workflows, AWS Step Functions, Azure Data Factory, or similar technologies. - Experience implementing CI/CD, automated testing, version control, Infrastructure as Code, and platform automation. - Strong understanding of data governance, metadata management, lineage, quality frameworks, privacy controls, and access management. - Experience diagnosing and resolving complex performance, reliability, scalability, and operational challenges. - Ability to make sound architectural decisions while balancing delivery speed, innovation, maintainability, security, and cost. - Proven ability to thrive in R&D and innovation-focused environments where priorities and requirements may evolve through experimentation. - Strong communication, collaboration, technical leadership, and mentoring skills. - Bachelor's degree in Computer Science, Engineering, Information Systems, or a related technical discipline, or equivalent practical experience. **Preferred Qualifications** - Experience supporting machine learning, generative AI, or agentic AI systems. - Experience with MLflow or comparable ML lifecycle tooling. - Experience with Azure Machine Learning, Azure AI Foundry, or Microsoft Fabric. - Experience with Unity Catalog, Microsoft Purview, or comparable catalog and governance platforms. - Experience with Kafka or other event-streaming technologies. - Experience with containerized and cloud-native platforms, including Docker and Kubernetes. - Experience with Terraform or another Infrastructure as Code framework. - Experience integrating structured, semi-structured, unstructured, and streaming data. - Knowledge of responsible AI, model evaluation, and AI platform observability. - Experience in payments, financial services, or another regulated industry. For more fresh new jobs, check out [yubhub.co](https://yubhub.co/). ## Skills ### Required - Python - PySpark - Apache Spark - SQL - AWS - Azure - Databricks - Apache Airflow - Data Architecture - Machine Learning - Generative AI - Agentic AI ### Nice to have - MLflow - Azure Machine Learning - Unity Catalog - Microsoft Purview - Kafka - Docker - Kubernetes - Terraform --- Source: [Apply at mastercard.wd1.myworkdayjobs.com](https://mastercard.wd1.myworkdayjobs.com/en-US/CorporateCareers/job/Pune-India/Principal-Data-Engineer--AWS--Databricks--Ai--ML-Flow--Data-Architecture--Apache-Airflow-_R-291404?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)