Description
We are seeking a Lead Data Engineer to join our AI&DPE Data Engineering platform team. As a Lead Data Engineer, you will be responsible for building and evolving large-scale backend data systems, real-time and batch pipelines, and AI-enabled services that power analytics, decisioning, and automation across the organization.
Primary Responsibilities
- Design, develop, and maintain backend services and data pipelines using Scala, Python, and Java
- Build and optimize batch and streaming workloads on Cloudera Data Platform (CDP) using Spark
- Work with Cloudera Manager to support platform configuration, monitoring, performance tuning, and operational stability
- Design and implement high-quality datamarts and curated datasets with strong emphasis on data integrity, performance, and reliability
- Design, build, and integrate AI-powered agents that operate to support anomaly detection and operational intelligence workflow automation
- Apply generative AI-driven or rule-based agents to reduce manual effort and improve scalability across backend systems
- Build and support data and compute workloads in AWS environments
- Leverage Databricks for large-scale data processing and advanced analytics
- Contribute to cloud-native and hybrid architectures integrating on-prem and cloud platforms
- Enable downstream consumers through well-designed, reliable backend data interfaces
- Partner with analytics, fraud, and business teams to ensure backend systems meet evolving needs without compromising platform stability
- Create clear technical documentation, including architecture diagrams, data flows, and design specifications
- Participate in Agile/Scrum ceremonies and cross-functional design reviews
- Mentor and upskill team members in backend engineering, big data, and AI agent concepts
Knowledge and Skill Requirements
- BS/BA degree in Computer Science, Engineering, Information Systems, or related field
- Strong hands-on experience with Scala, Python, Java in backend or data-intensive systems
- Experience working with Cloudera Data Platform (CDP) and Spark
- Familiarity with Cloudera Manager for cluster administration, monitoring, or troubleshooting
- Strong understanding of data modeling concepts, distributed systems, and large-scale data processing
- Excellent problem-solving skills and ability to work independently in complex environments