Description
As an Associate Developer, Data Engineering, you will be part of the Data & Analytics Platform Engineering Team, designing, developing, testing, and supporting various platform foundational frameworks in Amazon Web Services (AWS) and Databricks.
You will make an impact by:
- Designing, developing, and maintaining scalable data pipelines with delta mechanism in the AWS Lake using AWS services and PySpark.
- Analysing, designing, developing, testing, deploying, and supporting data models in AWS S3 using Spark SQL based on extracts from various sources and functional design specifications.
- Building and administering data ingestion frameworks in Databricks using PySpark and SQL and managing Platform Engineering tasks.
- Reviewing business requirements and creating functional and technical design documents, data lineage, data dictionaries, and metadata management, including SQL scripting.
- Staying current with industry trends and best practices in data engineering, Databricks, AWS services, AWS Lakehouse, PySpark, and Athena while participating actively in local and global teams.
To set you up for success in this role from day one, 3M requires (at a minimum) the following qualifications:
- Bachelor’s degree or higher (completed and verified prior to start).
- Two (2) to four (4) years of solid experience in data warehousing, including data pipelines and data modeling, particularly on AWS and Databricks platforms, in a private, public, government or military environment.
Additional qualifications that could help you succeed even further in this role include:
- Two (2) to three (3) years of strong experience in Python and PySpark with SQL scripting, plus hands-on experience with AWS services such as S3, Amazon EKS, Amazon EMR, CloudFormation, AWS Secrets Manager, Glue, and Athena.
- Two (2) to three (3) years of solid experience with Databricks, including automated user provisioning, RBACs, ABACs, AI/ML, and workspace management, along with good experience in relational databases and concepts.
- Extensive working knowledge of one or more business processes such as sales, finance, supply chain, manufacturing, or Master Data.
- Detail-oriented, self-directed problem solver with strong analytical and multitasking skills, a strong interest in technology, and English fluency.