Description
We are seeking a skilled and proactive Data Platform Administrator to manage, monitor, and optimize our cloud data platform environment. The ideal candidate will have hands-on experience with cloud-based data platforms, strong knowledge of Spark, big data or related technologies, and a passion for enabling data teams to work efficiently and securely.
Key Responsibilities
- Platform Administration: Handle and maintain workspaces, clusters, pools, and jobs in Databricks or other big data systems. Monitor performance, usage, and costs; optimize cluster configurations.
- Security & Compliance: Configure role-based access control (RBAC), and audit logging. Ensure compliance with data governance and security standards.
- Automation & DevOps: Develop CI/CD pipelines for notebooks, jobs, and libraries using tools like Gitlab, GitHub Actions, Azure DevOps, or Jenkins. Automate cluster lifecycle management and job scheduling.
- Support & Troubleshooting: Provide support to the users of the platform i.e., data scientists, engineers, and analysts. Troubleshoot job failures, performance bottlenecks, and integration issues.
- Integration & Collaboration: Integrate Databricks with other enterprise systems (e.g., Azure Data Lake, Power BI, Dataverse). Collaborate with data engineers and other stakeholders.
Required Qualifications
- Proven experience administering big data platforms in a production environment.
- Strong understanding of Apache Spark and distributed computing.
- Experience with cloud platforms (Azure preferred, AWS or GCP acceptable).
- Familiarity with Unity Catalog, Delta Lake, and Lakehouse architecture.
- Proficiency in Python, SQL, and shell scripting.
- Experience with CI/CD tools and infrastructure-as-code (Terraform is a plus).
- Excellent problem-solving and communication skills.
Benefits
- Healthcare
- Retirement planning
- Paid volunteering days
- Wellbeing initiatives