Description
This job will design, develop, and maintain data pipelines and architectures to support business needs.
You will work with stakeholders to understand data requirements and deliver solutions. Your role involves ensuring data quality, optimizing data processes, and collaborating with cross-functional teams.
Essential Responsibilities:
- Design and develop data pipelines for efficient data collection and processing.
- Build and maintain scalable data storage solutions.
- Ensure data quality and consistency through validation and cleansing processes.
- Implement data transformation techniques to prepare data for analysis.
- Collaborate with cross-functional teams to understand data requirements and provide solutions.
- Optimize data engineering processes for performance and reliability.
Your way to impact Your pipelines and data models will turn raw data from many systems into clean, scalable datasets that teams across PayPal rely on every day. You will help reduce duplicated work, improve data quality and speed up how quickly new metrics reach the business. Working closely with senior engineers, analysts and product partners, you will see your work used directly in reporting and analytics.
Your day to day In your day to day role you will:
- Write clean, solid and well-tested code and SQL for data pipelines that scale to millions of customers.
- Work with a team of talented engineers, product managers and architects to design and develop a highly scalable, configurable data platform on Google Cloud.
- Take part in design meetings to propose, discuss and refine new features and functionality.
- Apply business knowledge and technical understanding of application, information and data architectures to build the right solutions.
- Contribute across the full project life cycle, from kickoff and requirements analysis through design, implementation and release.
- Work independently and as part of the team, and communicate clearly with business and technology partners on technical standards, approaches and progress.
- Analyse problems, suggest innovative solutions and write detailed technical specifications.
- Collaborate effectively with cross-functional groups to deliver results.
- Support the platform through the product development life cycle (PDLC), including testing, deployment and production support.
- Maintain and improve existing architecture and design documentation, and create new documentation as needed.
Tech stack: Google Cloud Platform (BigQuery, Cloud Storage), Python, SQL, gcloud / gsutil / bq, plus Airflow, Hadoop, Spark and Hive.