Description
Visa is looking for a Staff Data Engineer to join their team. As a Staff Data Engineer, you will be responsible for designing, implementing, and improving software applications and systems. You will work hands-on with code, data, and modern tools, including AI-assisted development, cloud services, and automation frameworks, to deliver secure, scalable, and high-quality technology solutions that drive business outcomes in the fintech sector.
Key Responsibilities:
- Implement the transition from SQL Server data warehouse to the next-generation Hadoop/Databricks platform, ensuring performance, reliability, and minimal business disruption.
- Develop hybrid data pipelines that bridge legacy and modern ecosystems, enabling near real-time data access for analytics and AI applications.
- Optimize existing SQL Server models and design modern equivalents in Hadoop and Spark environments.
- Define long-term migration strategy, data partitioning, and retention policies aligned with Visa's data governance standards.
- Implement scalable, distributed data pipelines using Spark, Kafka, Airflow, and Delta Lake.
- Build robust ETL/ELT frameworks to process transactional, behavioral, and unstructured data at scale.
- Partner with the Agentic AI team to power RAG pipelines, vector database integrations, and LLM data provisioning.
- Lead proof-of-concept initiatives to evaluate and integrate new data engineering technologies.
- Provide guidance to junior Engineers to maintain existing SQL Server-based data warehouses.
- Implement best practices in T-SQL development, schema design, and stored procedure optimization.
- Perform proactive performance analysis, troubleshooting, and resolution of production issues in SQL Server and Hadoop clusters.
- Collaborate with DBA and application teams to ensure uptime within SLAs and compliance with Visa's data policies.
- Enforce standards for data quality, lineage, and governance across both legacy and modern platforms.
- Ensure full compliance with Visa's data privacy, encryption, and access control frameworks.
- Partner with InfoSec to embed data security principles into every phase of data lifecycle management.
- Mentor and coach junior data engineers, fostering an environment of technical excellence and innovation.
- Collaborate cross-functionally with product, ML, and platform engineering teams on architecture decisions and roadmap execution.
Requirements:
- Bachelor's or Master's degree in Computer Science, Data Engineering, or related field.
- 8+ years of hands-on data engineering experience, including enterprise-scale data warehousing and pipeline design.
- Proven success supporting and modernizing SQL Server-based data warehouses in high-SLA environments.
- Production-level experience architecting Hadoop, Spark, and Databricks data pipelines.
- Expertise in ETL/ELT frameworks, data modeling, and schema design for analytical and operational use cases.
- Strong programming proficiency in Python, Java, or Scala.
- Hands-on experience with AWS or Azure (Glue, Synapse, Redshift, Delta Lake, S3).
- Familiarity with Kafka, Airflow, Kubernetes, and containerized data services.
- Understanding of RAG pipelines, vector databases, and AI data flows.
- Experience designing data pipelines and APIs compatible with Model Context Protocol (MCP)-based agent frameworks.
- Strong SQL optimization, debugging, and production troubleshooting experience.
Preferred Qualifications:
- Experience developing data warehouse migration or modernization from SQL Server to Hadoop/Spark ecosystems.
- Deep understanding of data lineage, metadata management, and governance frameworks.
- Familiarity with LangChain, LangGraph, and MCP for integrating AI agents with data systems.
- Strong ability to balance innovation and stability across coexisting legacy and modern data architectures.
- Proven track record mentoring engineers and collaborating across global teams.
Tech Stack Snapshot:
- Languages: Python, Java, Scala, SQL, T-SQL
- Data Systems: SQL Server, Hadoop, Spark, Databricks, Delta Lake, Snowflake
- Pipelines: Kafka, Airflow, Glue, Azure Data Factory
- AI Integration: LangChain, LangGraph
- Cloud Platforms: AWS, Azure
- Infra: Docker, Kubernetes, Terraform, Jenkins
- Governance: Great Expectations, Atlas, Data Privacy & Encryption Frameworks