Description
We are seeking an Enterprise Data Platform Architect to design and build a next-generation, vendor-agnostic Data Platform from the ground up. You will own the foundational platform architecture,decoupling compute from storage using Apache Iceberg, designing containerized execution engines on Kubernetes, establishing multi-engine processing, and embedding automated governance from Day 1.
Key Responsibilities:
- Custom Platform Architecture: Lead design decisions for a decoupled Lakehouse architecture on AWS and Kubernetes, establishing storage layout, partition evolution, and open table formats.
- Multi-Engine Compute Optimization: Match specific workloads to appropriate engines to maximize speed while minimizing cloud spend.
- Framework & Runner Engineering: Define config-driven pipeline patterns and containerized execution runners that automatically handle idempotency, schema evolution, advisory locking, and step-level logging.
- Embedded Governance & Cataloging: Architect automated, zero-trust data governance from Day 1, integrating open-source catalogs with column-level security, PII discovery, and automated audit trails.
- Data Quality Frameworks: Design multi-layer, config-driven data quality gates that execute inline in-memory and trigger automated pipeline halts before bad data propagates.
- Architectural Decision Records: Maintain a version-controlled repository of technical decisions, documenting trade-offs, compute boundaries, and long-term implications.
Core Technical Qualifications:
- 8+ Years in Platform Architecture / Infrastructure Engineering: Proven track record of designing and deploying production-grade, open-standard Data Platforms from scratch.
- Open Table Formats & Decoupled Storage: Deep expertise in Apache Iceberg, including native PyIceberg APIs, ACID transactions on object storage, schema/partition evolution, and metadata management.
- High-Efficiency & Embedded Compute: Hands-on experience with in-process/vectorized engines and matching workload scales to the right compute engine.
- Kubernetes & Container Orchestration: Experience in Kubernetes, specifically Pods, Deployments, ConfigMaps, Secrets, and custom containerized execution runners.
- Advanced Orchestration: Expert-level Apache Airflow skills, specifically designing dynamic, dependency-aware DAGs and managing worker architecture in resource-constrained environments.
- Transformation & Semantic Layers: Deep proficiency with dbt Core paired with external execution engines and semantic YAML layer definitions.
- Open-Source Catalogs & Governance: Experience implementing open catalog specifications and metadata platforms.
- Cloud Infrastructure: Advanced knowledge of AWS core services with strict adherence to storage-compute separation and FinOps spend optimization.
What We Offer: At TaskUs, we prioritize our employees' well-being by offering competitive industry salaries and comprehensive benefits packages. Our commitment to a People First culture is reflected in the various departments we have established, including Total Rewards, Wellness, HR, and Diversity.