Description
Husky is a distributed, petabyte-scale columnar event store at the heart of Datadog's Event Platform, which powers dozens of popular products.
As a Senior Software Engineer on this team, you'll work hands-on with a distributed storage system at petabyte scale, and help evolve the platform to support the next level of scale that will power the AI era.
You'll partner closely with Husky Ingestion, Metadata, and Platform, and increasingly with Lakehouse on the Iceberg-compatible boundary between Husky and analytical/batch workloads.
Responsibilities:
- Drive Husky to the next level of adoption by migrating the storage format to open source Parquet and re-imagine compaction on a format-agnostic, DataFusion-based substrate
- Drive concrete cost-reduction work: block-level inverted index compression, combining the database with external objects at the row level, and storage tiering/caching
- Scale the solution to enable customers to manage event versioning at scale, using temporal-database techniques
- Play an influential role in operating and deciding how to best operate the service going forward
- Work across the write path (with Ingestion) and toward a cleaner Iceberg-based handoff to ADP/Lakehouse for batch/analytical workloads
Requirements:
- Enthusiastic and experienced with large-scale distributed systems and/or storage engines
- Comfortable with improving and operating large distributed systems that customers rely on for critical business functions
- Strong fundamentals in columnar/file storage formats (Parquet, ORC, Arrow) or comparable systems
Benefits:
- New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
- Continuous professional development, product training, and career pathing
- Intradepartmental mentor and buddy program for in-house networking
- An inclusive company culture, ability to join Community Guilds (Datadog employee resource groups)
- Free, global mental health benefits for employees and dependents age 6+
The reasonably estimated yearly salary for this role at Datadog is: $192,000-$240,000 USD