Vice President - LME Data Platform

We are seeking a Vice President to lead the architecture, design, and technical strategy for our open-source data platform. You will make the end-to-end technical vision , from ingestion to analytics , while mentoring a team of data platform engineers. This is a hands-on leadership role responsible for designing blueprints and writing critical-path code.

Responsibilities

Architecture & Strategy

• Define the overall data platform architecture on OpenShift, covering ingestion (Kafka/Redpanda), compute (Spark, Flink, Trino), storage (MinIO, Iceberg), catalog (Polaris/Gravitino), orchestration (Airflow/Dagster), and serving (StarRocks/ClickHouse).
• Design the Data Lakehouse architecture: open table formats (Iceberg, Delta Lake), catalog federation, multi-engine interoperability.
• Architect the real-time data path: Kafka topic design, schema registry (Apicurio/Confluent), stream processing (Flink/Spark Structured Streaming).
• Design the analytical serving layer: OLAP engine selection (StarRocks/ClickHouse/Doris), materialized view strategy, query federation with Trino.
• Design multi-tenancy, data security (Ranger/OpenPolicyAgent), RBAC, and governance (DataHub/Atlas) across all platform layers.
• Set technical standards for Helm chart design, CI/CD pipelines, and GitOps (ArgoCD/Flux) workflows.

Technical Leadership

• Lead a team of 3–6 data platform engineers; run code reviews, design reviews, and sprint planning.
• Establish engineering best practices: testing, observability (OpenTelemetry, Prometheus, Grafana, Loki), incident response, runbooks.
• Partner with data engineering, analytics, and business teams to translate their needs into platform capabilities.
• Define platform SLOs/SLIs across freshness, latency, availability, and durability; drive the on-call rotation and incident post-mortems.

Hands-on Engineering

• Build and maintain the core OpenShift infrastructure: operators, Helm charts, namespaces, RBAC, network policies.
• Develop Spark and Flink job frameworks, tuning guides, workload scheduling (YuniKorn/Volcano).
• Implement the data mesh or data product model: domain ownership, self-serve data infrastructure, federated governance.
• Performance-optimize across the stack: Kafka throughput, Spark shuffle, Iceberg compaction, Trino query planning, OLAP cold reads.

Required Skills & Experience

• 10+ years in data/platform engineering, with 3+ years as architect or tech lead.
• Deep Kubernetes/OpenShift: operators, Helm, CRDs, admission webhooks, SCC, multi-tenancy at production scale.
• Expert-level streaming: Kafka/Redpanda internals (partitioning, consumer groups, exactly-once semantics), schema registry, Kafka Connect.
• Expert-level batch & streaming compute: Spark internals (Catalyst, Tungsten, AQE, shuffle, dynamic allocation), Flink or Spark Structured Streaming.
• Strong open table format knowledge: Iceberg and/or Delta Lake , table format internals, metadata evolution, partitioning transforms, maintenance, catalog integration (Polaris, Gravitino, Nessie, or Unity Catalog).
• Hands-on with federated query engines: Trino (coordinator/worker architecture, connector ecosystem, fault-tolerant execution mode).
• Solid OLAP knowledge: StarRocks, ClickHouse, or Apache Doris , primary key models, materialized views, query optimization.
• Production MinIO/S3-compatible storage at scale: erasure coding, multi-site replication, IAM, tiering.
• Pipeline orchestration: Airflow (DAG design, sensors, dynamic task mapping) or Dagster.
• Infrastructure-as-code (Terraform/Pulumi/Crossplane) and GitOps (ArgoCD/Flux).
• Fluent in at least one JVM language (Scala/Java) and Python.