Description
We are seeking a Data Platform Engineer to design, build, and operate critical subsystems of our data platform. You will take ownership of major components, drive them from design to production, and continuously improve their reliability and performance.
Responsibilities:
- Streaming Infrastructure: Manage Kafka cluster operations, including broker tuning, partition rebalancing, monitoring, and disaster recovery. Oversee schema registry and Kafka Connect connectors.
- Batch & Streaming Compute: Develop and optimize Spark batch jobs and Flink/Spark Structured Streaming pipelines. Create reusable job frameworks and tuning guides.
- Storage & Lakehouse: Maintain Iceberg tables, including compaction, snapshot expiration, and schema evolution. Operate MinIO at scale, configuring lifecycle rules, tiering, and performance tuning.
- Query & Serving: Deploy and manage Trino clusters, including connector configuration, resource groups, and query monitoring. Oversee StarRocks/ClickHouse clusters, focusing on sharding, replication, and materialized views.
- Orchestration: Build and maintain Airflow/Dagster DAGs for platform operations, extending custom operators and sensors.
- Platform Observability: Implement monitoring with OpenTelemetry, Prometheus, Grafana, and Loki across all stack layers. Develop dashboards and alerting rules.
- Kubernetes Operations: Author Helm charts, manage operator lifecycles, configure resource quotas, node affinity, and pod disruption budgets.
- CI/CD & GitOps: Own ArgoCD application sets, Helm-based deployments, and promotion pipelines from dev to prod.
- Participate in on-call rotation, writing post-mortems and runbooks.
- Mentor mid-level engineers through pairing, design discussions, and code reviews.
Required Skills & Experience:
- 6+ years of experience in data/platform engineering or backend infrastructure
- Strong Kubernetes skills, including Helm chart authoring, RBAC, network policies, storage, and operators
- Solid Kafka expertise, including topic design, consumer group management, offset management, monitoring lag, Kafka Connect, and schema registry
- Proficient in Spark, including Dataframe/Dataset API, Spark SQL, performance tuning, and troubleshooting in production
- Working knowledge of Flink or Spark Structured Streaming for real-time pipelines
- Practical Iceberg experience, including table maintenance, time-travel, and catalog integration
- Hands-on Trino or Presto experience, including connector configuration, query tuning, and resource group management
- Experience with an OLAP engine, such as StarRocks, ClickHouse, or Doris
- Proficient in Python and either Scala or Java
- Solid CI/CD and GitOps experience, including ArgoCD or Flux, Helm, and Docker
- Airflow or Dagster experience for pipeline orchestration