Description
We are seeking a Software Engineer III to join our Data Intelligence Team. As a Data Engineer, you will design, build, and optimize data pipelines and platforms that enable reliable, scalable, and secure data processing across enterprise environments.
Responsibilities:
- Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory, Databricks, and Azure storage services.
- Ingest data from multiple sources including relational databases, APIs, and file-based systems into Azure data platforms.
- Implement data transformations on HPCC using ECL and on Azure platform using SparkSQL and PySpark in Azure Databricks.
- Develop and optimize notebooks and jobs using Azure Databricks.
- Write efficient Spark jobs for large-scale data processing.
- Design and implement data models on HPCC using ECL and also using Delta Lake, Parquet, or other columnar storage formats.
- Optimize ECL code, SQL, and Spark workloads for performance and scalability.
- Work closely with business analysts, data scientists, and application teams to understand data requirements and promote data engineering best practices.
Requirements:
- Strong proficiency in SQL and data transformation logic with solid understanding of data warehousing concepts and ETL/ELT patterns.
- Hands-on experience with Azure Databricks, Spark SQL, and PySpark.
- Experience with Azure Data Factory, Azure Data Lake Storage (ADLS Gen2), and Azure SQL / Synapse.
- Experience working with large-scale, structured and semi-structured datasets.
- Experience with Agile development methodologies.
- Must have a positive attitude and desire to learn new processes and technologies.
- Knowledge of SDLC process like requirement gathering, design, implementation (Coding), testing, deployment, and maintenance.
- Excellent communication skills – verbal and written, Strong problem-solving skills with good knowledge of Insurance terminology.
We welcome applications from individuals of all backgrounds, including underrepresented groups, people with disabilities, career changers, and candidates with non-traditional education or experience.