Description
TaskUs is seeking a Senior Data Engineer to join their team. As a Senior Data Engineer, you will be the engine room of our data strategy, building resilient, self-healing systems that transform raw data into high-fidelity data products.
Responsibilities:
- Pipeline Engineering: Build and maintain high-throughput ETL/ELT pipelines that ingest data from various sources into our Lakehouse.
- Code Quality & Tooling: Drive the adoption of dbt for transformation and PySpark for heavy lifting, writing modular, reusable code that follows strict CI/CD practices.
- Observability & Reliability: Implement "Data SLAs," building monitoring and alerting systems that notify the team of data drift or pipeline failures before the business notices.
- Data Productization: Work closely with the BI team and Data Scientists to prepare "feature-ready" datasets.
- Performance Tuning: Deep-dive into SQL and Spark query plans to optimize slow-running jobs, reducing cost and latency.
- Local Development Advocacy: Implement a workflow where engineers can develop and test complex SQL transformations locally using DuckDB.
- Cost-Efficient Micro-Pipelines: Build specialized pipelines for specific client reporting or data-quality checks that run on single-node containers.
- Embedded Analytics: Explore ways to use DuckDB as an embedded engine for internal tools or "edge" processing within the BPO's local site offices.
Requirements:
- 5+ years in Data Engineering
- Expert-level proficiency in Python, SQL, and PySpark
- Modern Lakehouse Stack experience with Databricks (Delta Lake) or Snowflake
- Advanced experience with dbt (Data Build Tool)
- Experience with Apache Airflow or Prefect
- Familiarity with Kafka, Kinesis, or Spark Streaming
- Infrastructure as Code (IaC) experience with Terraform or CloudFormation
- Proven experience using DuckDB for high-speed local development or unit testing data transformations
- Hybrid Execution Patterns experience
- Parquet/Iceberg Interaction experience
- Orchestration experience with Kubernetes (K8s)
What We Offer:
- Competitive industry salaries
- Comprehensive benefits packages
- People First culture
- Internal mobility and professional growth opportunities