Description
We are looking for a Technical Architect to own the technical vision and architecture for distributed tracing as a first-class pillar of Workday's Observability Platform.
The Data Platform and Observability Engineering (DPOE) team is building Workday's next-generation, multi-petabyte scale Observability Platform.
Responsibilities:
- Architect and build Workday's distributed tracing platform on ClickHouse/Tempo, designed for multi-petabyte scale ingestion and sub-second interactive query performance.
- Own the big-data pipeline feeding tracing data , Kafka-based ingestion, Spark/Flink stream and batch processing, and Iceberg-on-S3 storage , including schema design, partitioning, compaction, and lifecycle management.
- Drive performance and scaling across ingestion and query paths: storage format optimization (Parquet/Iceberg), compression strategy, partitioning/indexing, and query engine tuning under real production load.
- Lead HA/DR design for tracing services , multi-region/multi-AZ resilience, failover, backup/restore, and recovery time/point objectives appropriate to a tier-1 platform.
- Design security architecture for the platform, including authentication/authorization (authn/authz) for multi-tenant data access across ingestion and query layers.
- Own operational excellence for distributed tracing: monitoring, logging, alerting, capacity planning, and participation in an on-call rotation for the platform.
- Evaluate and introduce new technologies , open source and cloud-native , that materially improve the platform's scalability, cost efficiency, or capability.
- Shape the future of Observability AI: Extend distributed tracing to LLM and agentic workflows using LangSmith and LangChain, enabling observability into multi-step agent execution, tool calls, and prompt/response chains.
- Evangelize the platform: publish best practices, mentor engineers across DPOE and partner teams, and act as a technical thought leader for the modern observability/data stack internally.
- Operate with high autonomy in a fast-moving, ambiguous environment , setting technical direction with minimal oversight while aligning with broader platform strategy.
Requirements:
- 14+ years experience in software development engineering.
- 6+ years experience specifically focused on designing, building, and operating complex distributed system architectures.
- 8+ years experience with at least two of the following programming languages (e.g., Java, Python, Go).
- Bachelor’s degree in a relevant field such as Computer Science, Engineering, or a related discipline; a Master's degree (e.g., MS in Computer Science, Distributed Systems, or related field) is strongly preferred or equivalent practical experience.
Workday Pay Transparency Statement: The annualized base salary ranges for the primary location and any additional locations are listed below. Primary Location: USA.CA.Pleasanton Primary Location Base Pay Range: $222,900 USD - $334,300 USD Additional US Location(s) Base Pay Range: $187,100 USD - $334,300 USD