Description
We are seeking a skilled and proactive Senior Data Platform Administrator to handle, monitor, and optimize our cloud data platform environment.
The ideal candidate will have hands-on experience with cloud-based data platforms, strong knowledge of Spark, big data or related technologies, and a passion for enabling data teams to work efficiently and securely.
Key Responsibilities:
- Administer, maintain, and optimize Databricks and cloud-based data platform environment Azure, including workspaces, clusters, jobs, and instance pools.
- Ensure platform reliability, availability, and performance through proactive monitoring, alerting, capacity planning, and operational excellence practices.
- Monitor and optimize platform utilization and cloud costs while maintaining performance and scalability.
- Implement and handle security controls, RBAC, audit logging, and compliance with enterprise governance standards.
- Develop and maintain Infrastructure-as-Code (Terraform) and CI/CD pipelines using DevOps tools, GitHub Actions, or GitLab.
- Automate platform provisioning, cluster lifecycle management, job orchestration, and routine operational tasks.
- Provide production support for data engineers, data scientists, and analysts, ensuring timely resolution of incidents and service requests.
- Troubleshoot platform issues, Spark workloads, job failures, performance bottlenecks, and integration challenges.
- Establish and maintain SRE practices, including SLAs, SLOs, error budgets, incident management, root cause analysis, and continuous service improvement.
- Design and maintain observability solutions, including monitoring dashboards, logging, and alerting frameworks.
- Support high availability, disaster recovery, backup, patching, and platform upgrade activities.
- Integrate Databricks with enterprise systems and services, including Azure Data Lake, Power BI, Dataverse, and other cloud-native technologies.
- Collaborate with data engineering, architecture, security, and operations teams to deliver secure, scalable, and resilient data platform solutions.
- Support Lakehouse architecture capabilities, including Unity Catalog, Delta Lake, governance, and data management best practices.
Required Qualifications:
- Validated experience administering big data platforms in a production environment.
- Strong understanding of Apache Spark and distributed computing.
- Experience with cloud platforms (Azure preferred).
- Familiarity with Unity Catalog, Delta Lake, and Lakehouse architecture.
- Proficiency in Python, SQL, and shell scripting.
- Experience with CI/CD tools and infrastructure-as-code (Terraform is a must).
- Excellent problem-solving and interpersonal skills