Description
The Senior Analyst, Data Integration & Workflows plays a critical role within the Data AI & Enablement organization, serving as a senior technical leader responsible for designing, implementing, and operationalizing production-grade data pipelines and workflow automation that power SPDJI's index and analytical solutions.
Responsibilities and Impact:
- Technical Leadership & Solution Delivery:
- Lead complex data integration initiatives from design through production deployment, ensuring solutions are scalable, observable, and aligned with enterprise architecture standards.
- Design and implement production-grade data pipelines (batch and streaming) that transform raw inputs into trusted curated outputs.
- Establish and evangelize engineering best practices for ETL/ELT patterns, workflow orchestration, data quality controls, and operational observability.
- Drive technical decision-making for pipeline architecture, technology selection, and design patterns.
- Enablement & Co-Development:
- Lead hands-on enablement with value stream SMEs through pair programming, structured guidance, and co-development sessions.
- Assess SME technical readiness and recommend appropriate engagement models.
- Build reusable automation components and templates that accelerate consistent delivery across domains.
- Develop SME technical capabilities through targeted coaching, code reviews, and knowledge transfer.
- Quality Assurance & Production Readiness:
- Conduct comprehensive code reviews for SME-built and team-developed pipelines.
- Implement data reliability controls including validation rules, reconciliation checks, anomaly detection, and completeness/timeliness monitoring.
- Engineer observability and monitoring solutions by implementing logging standards, metrics, alerts, and runbooks.
- Operational Excellence & Continuous Improvement:
- Provide L3 support for production business-logic issues, collaborating with value stream SMEs to drive root-cause analysis and implement permanent fixes.
- Optimize pipeline performance and cost through appropriate partitioning strategies, caching, incremental processing patterns, and compute resource tuning.
- Capture and share lessons learned, updating engineering playbooks, patterns, and standards based on production outcomes and emerging best practices.
- Collaboration & Stakeholder Management:
- Collaborate with Data Integration Lead to shape team strategy, prioritize initiatives, and align technical approaches with organizational goals.
- Partner effectively with AI Solutions and Data Governance teams on cross-cutting concerns.
- Engage with Data Value Streams to understand business requirements, validate technical solutions, and ensure alignment with domain expertise.
Requirements:
- Experience:
- 8+ years of experience in data engineering, data integration, or related technical roles with progressive responsibility.
- 3+ years of experience in technical leadership roles, including mentoring engineers and leading complex technical initiatives.
- Technical Expertise:
- Proficiency in ETL/ELT design patterns, data pipeline architecture, and workflow orchestration frameworks.
- Advanced programming skills in languages commonly used in data engineering (Python, SQL, Scala, or similar).
- Solid understanding of data quality frameworks, data validation techniques, reconciliation patterns, and anomaly detection.
- Leadership & Soft Skills:
- Strong technical leadership with demonstrated ability to lead complex initiatives and influence technical direction.
- Excellent mentorship and coaching abilities, with track record of developing technical talent and improving team capabilities.
- Outstanding collaboration skills with ability to partner effectively across technical and business teams in a matrixed organization.