Description
As a Software Engineer III, you will design, build, and optimize data pipelines and platforms that enable reliable, scalable, and secure data processing across enterprise environments. You will work with modern Azure technologies and collaborate with business analysts, data scientists, and application teams to deliver high-quality data solutions that support analytics, reporting, and operational needs.
Responsibilities:
- Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory, Databricks, and Azure storage services.
- Ingest data from multiple sources including relational databases, APIs, and file-based systems into Azure data platforms.
- Implement data transformations on HPCC using ECL and on Azure platform using SparkSQL and PySpark in Azure Databricks.
- Develop and optimize notebooks and jobs using Azure Databricks.
- Write efficient Spark jobs for large-scale data processing.
- Design and implement data models on HPCC using ECL and also using Delta Lake, Parquet, or other columnar storage formats.
- Optimize ECL code, SQL, and Spark workloads for performance and scalability.
- Work closely with business analysts, data scientists, and application teams to understand data requirements and promote data engineering best practices.
Requirements:
- Strong proficiency in SQL and data transformation logic with solid understanding of data warehousing concepts and ETL/ELT patterns.
- Hands-on experience with Azure Databricks, Spark SQL, and PySpark.
- Experience with Azure Data Factory, Azure Data Lake Storage (ADLS Gen2), and Azure SQL / Synapse.
- Experience working with large-scale, structured and semi-structured datasets.
- Experience with Agile development methodologies.
- Must have a positive attitude and desire to learn new processes and technologies.
- Knowledge of SDLC process like requirement gathering, design, implementation (Coding), testing, deployment, and maintenance.
- Excellent communication skills – verbal and written, Strong problem-solving skills with good knowledge of Insurance terminology.