# Technical Architect - Distributed Tracing

**Company**: Workday
**Location**: Pleasanton, CA
**Work arrangement**: hybrid
**Experience**: senior
**Job type**: full-time
**Salary**: $222,900 USD - $334,300 USD
**Category**: Engineering
**Industry**: Technology

**Apply**: https://workday.wd5.myworkdayjobs.com/en-US/Workday/job/USA-CA-Pleasanton/Principal-Distributed-Systems-Engineer---Observability_JR-0109636?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_69974466-8c0

## Description

We are looking for a Technical Architect to own the technical vision and architecture for distributed tracing as a first-class pillar of Workday's Observability Platform.

The Data Platform and Observability Engineering (DPOE) team is building Workday's next-generation, multi-petabyte scale Observability Platform.

Responsibilities:

- Architect and build Workday's distributed tracing platform on ClickHouse/Tempo, designed for multi-petabyte scale ingestion and sub-second interactive query performance.

- Own the big-data pipeline feeding tracing data , Kafka-based ingestion, Spark/Flink stream and batch processing, and Iceberg-on-S3 storage , including schema design, partitioning, compaction, and lifecycle management.

- Drive performance and scaling across ingestion and query paths: storage format optimization (Parquet/Iceberg), compression strategy, partitioning/indexing, and query engine tuning under real production load.

- Lead HA/DR design for tracing services , multi-region/multi-AZ resilience, failover, backup/restore, and recovery time/point objectives appropriate to a tier-1 platform.

- Design security architecture for the platform, including authentication/authorization (authn/authz) for multi-tenant data access across ingestion and query layers.

- Own operational excellence for distributed tracing: monitoring, logging, alerting, capacity planning, and participation in an on-call rotation for the platform.

- Evaluate and introduce new technologies , open source and cloud-native , that materially improve the platform's scalability, cost efficiency, or capability.

- Shape the future of Observability AI: Extend distributed tracing to LLM and agentic workflows using LangSmith and LangChain, enabling observability into multi-step agent execution, tool calls, and prompt/response chains.

- Evangelize the platform: publish best practices, mentor engineers across DPOE and partner teams, and act as a technical thought leader for the modern observability/data stack internally.

- Operate with high autonomy in a fast-moving, ambiguous environment , setting technical direction with minimal oversight while aligning with broader platform strategy.

Requirements:

- 14+ years experience in software development engineering.

- 6+ years experience specifically focused on designing, building, and operating complex distributed system architectures.

- 8+ years experience with at least two of the following programming languages (e.g., Java, Python, Go).

- Bachelor’s degree in a relevant field such as Computer Science, Engineering, or a related discipline; a Master's degree (e.g., MS in Computer Science, Distributed Systems, or related field) is strongly preferred or equivalent practical experience.

Workday Pay Transparency Statement: The annualized base salary ranges for the primary location and any additional locations are listed below. Primary Location: USA.CA.Pleasanton Primary Location Base Pay Range: $222,900 USD - $334,300 USD Additional US Location(s) Base Pay Range: $187,100 USD - $334,300 USD

## Skills

### Required
- Distributed Systems
- API Development
- Algorithmic Thinking
- Large Scale Data Processing
- System Security

### Nice to have
- LangSmith
- LangChain
- ClickHouse
- Grafana Tempo
- Kafka
- Spark/Flink
- Iceberg
- S3

---

Source: [Apply at workday.wd5.myworkdayjobs.com](https://workday.wd5.myworkdayjobs.com/en-US/Workday/job/USA-CA-Pleasanton/Principal-Distributed-Systems-Engineer---Observability_JR-0109636?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)