Description
TaskUs is seeking a Senior Data Engineer to join their team. As a Senior Data Engineer, you will be the engine room of our data strategy, building resilient, self-healing systems that transform raw data into high-fidelity data products.
You will take the Architect's blueprints and turn them into production-grade code, ensuring there is clean and reliable data.
Responsibilities:
- Pipeline Engineering: Build and maintain high-throughput ETL/ELT pipelines that ingest data from various sources into our Lakehouse.
- Code Quality & Tooling: Drive the adoption of dbt for transformation and PySpark for heavy lifting. You will be responsible for writing modular, reusable code that follows strict CI/CD practices.
- Observability & Reliability: Implement "Data SLAs." You will build the monitoring and alerting systems that notify the team of data drift or pipeline failures before the business notices.
- Data Productization: Work closely with the BI team and Data Scientists to prepare "feature-ready" datasets.
- Performance Tuning: Deep-dive into SQL and Spark query plans to optimize slow-running jobs, reducing cost and latency.
- Local Development Advocacy: Implementing a workflow where engineers can develop and test complex SQL transformations locally using DuckDB, drastically reducing "waiting-for-cluster" time and lowering development costs.
- Cost-Efficient Micro-Pipelines: Building specialized pipelines for specific client reporting or data-quality checks that run on single-node containers (e.g., Lambda or small ECS tasks) using DuckDB, avoiding the minimum billing cycles of larger warehouses.
- Embedded Analytics: Exploring ways to use DuckDB as an embedded engine for internal tools or "edge" processing within the BPO's local site offices where bandwidth to the cloud might be a constraint.
Requirements:
- 5+ Years in Data Engineering
- Expert-level proficiency in Python, SQL, and PySpark
- Modern Lakehouse Stack: Hands-on experience with Databricks (Delta Lake) or Snowflake
- Transformation & Modeling: Advanced experience with dbt (Data Build Tool)
- Orchestration: Experience with Apache Airflow or Prefect
- Streaming Experience: Familiarity with Kafka, Kinesis, or Spark Streaming
- Infrastructure as Code (IaC): Comfort with Terraform or CloudFormation
- In-Process Analytics (DuckDB): Proven experience using DuckDB for high-speed local development, unit testing data transformations, or as a query engine for "small-to-medium" datasets
- Hybrid Execution Patterns: Ability to identify when to use heavyweight compute (PySpark/Databricks) versus lightweight compute (DuckDB/Python) to minimize cloud costs and reduce job latency.
What We Offer: At TaskUs, we prioritize our employees' well-being by offering competitive industry salaries and comprehensive benefits packages. Our commitment to a People First culture is reflected in the various departments we have established, including Total Rewards, Wellness, HR, and Diversity.
For more fresh new jobs, check out yubhub.co.