Description
We are seeking a Data Platform Operations & Observability Engineer to enhance and support our analytics platform across on-premises and Azure environments.
This role requires reliable support for on-premises and Azure analytics platforms through pipeline monitoring, incident resolution, automation, and observability improvements.
The engineer will investigate production issues, restore failed processes, communicate with stakeholders, and help improve platform reliability.
Over time, the engineer will also build and enhance monitoring, alerting, logging, dashboards, and operational processes to improve observability and reduce incidents.
Responsibilities
Operations & Support
- Monitor and support production data pipelines and analytics platforms and Identify opportunities to improve reliability, performance, and operational efficiency.
- Troubleshoot pipeline failures, data issues, application errors, and performance problems, data movement across on-premises and cloud environments.
- Perform root cause analysis and implement corrective actions and Communicate incidents, remediation plans, and service impacts to stakeholders.
- Support Python automation processes, scheduled jobs, and file transfer workflows.
- Administer Windows servers, including patching, software installation, vulnerability remediation, and system maintenance.
- Support Azure services including Databricks, Data Factory, Synapse, and Data Lake Storage.
- Create and maintain operational documentation, diagrams, and support procedures.
Observability & Reliability
- Design and maintain monitoring, alerting, and dashboard solutions.
- Build and maintain centralized logging and operational reporting.
- Define and track operational metrics, KPIs, SLAs, and SLOs and develop proactive monitoring to detect issues before they affect users.
- Automate operational tasks and repetitive support activities drive continuous improvements in platform reliability and observability.
Required Skills & Qualifications
- Strong Python development and troubleshooting skills with Pyspark also expected with Observability is considered the key requirement, including logging, monitoring, dashboard creation from logs, alerting, debugging, and performance analysis.
- Have relevant experience in cloud logging tools such as Grafana and Elasticsearch/Kibana, Azure Log Analytics Workspace and Amazon CloudWatch Logs.
- Experience working with Azure or other cloud platforms also and using common data formats such as CSV, JSON, XML, and Parquet.
- Experience administering Windows Server environments, Knowledge of data integration concepts, file transfers, APIs, and data validation.
- Understanding of networking fundamentals including DNS, proxies, firewalls, VPNs, and SSL/TLS.
- Strong troubleshooting and analytical skills, verbal, and written communication skills with ability to manage multiple priorities in a fast-paced environment, Self-motivated, detail-oriented, and eager to learn.