Senior Consultant Specialist

We are currently seeking an experienced professional to join our team in the role of Senior Consultant Specialist.

Principal Responsibilities

• Design and Develop ETL Processes: Lead the design and implementation of ETL processes using various batch/streaming tools to extract, transform, and load data from multiple sources into GCP. Utilise ETL technologies such as Control-M, Datastage, Shell Scripting, and Python for cloud migration projects.
• Collaborate with stakeholders to gather requirements and ensure ETL solutions meet business needs.

• Data Pipeline Optimization: Optimise data pipelines for performance, scalability, and reliability, ensuring efficient data processing workflows.
• Monitor and troubleshoot ETL processes, proactively addressing issues and bottlenecks.

• Data Integration and Management: Integrate data from diverse sources, including databases, APIs, and flat files, ensuring data quality and consistency.
• Manage and maintain data storage solutions in GCP (e.g., BigQuery, Cloud Storage) to support analytics and reporting.

• GCP Dataflow Development: Write Apache Beam-based Dataflow Jobs for data extraction, transformation, and analysis, ensuring optimal performance and accuracy.
• Collaborate with data analysts and data scientists to prepare data for analysis and reporting.

• Automation and Monitoring: Implement automation for ETL workflows using tools like Apache Airflow or Cloud Composer, enhancing efficiency and reducing manual intervention.
• Set up monitoring and alerting mechanisms to ensure the health of data pipelines and compliance with SLAs.

• Data Governance and Security: Apply best practices for data governance, ensuring compliance with industry regulations (e.g., GDPR, HIPAA) and internal policies.
• Collaborate with security teams to implement data protection measures and address vulnerabilities.

Requirements

• Education: Bachelor's degree in Computer Science, Information Systems, or a related field.
• Experience: Minimum of 10 years of industry experience in data engineering or ETL development, with a strong focus on Data Stage and GCP.
• Proven experience in designing and managing ETL solutions, including data modelling, data warehousing, and SQL development.

• Technical Skills: Strong knowledge of GCP services (e.g., BigQuery, Dataflow, Cloud Storage, Pub/Sub) and their application in data engineering.
• Experience with cloud-based solutions, especially in GCP; cloud-certified candidates are preferred.
• Experience and knowledge of Big Data processing in batch mode and streaming mode, proficient in Big Data ecosystems (e.g., Hadoop, HBase, Hive, MapReduce, Kafka, Flink, Spark).
• Familiarity with Java and Python for data manipulation on Cloud/Big Data platforms.

• Analytical Skills: Strong problem-solving skills with a keen attention to detail.
• Ability to analyse complex data sets and derive meaningful insights.