# Principal Production Engineer

**Company**: Zscaler
**Location**: Santa Clara, California
**Work arrangement**: hybrid
**Experience**: senior
**Job type**: full-time
**Salary**: $164,500-$235,000 USD
**Category**: Engineering
**Industry**: Technology

**Apply**: https://job-boards.greenhouse.io/zscaler/jobs/5149607007?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_573d975d-f4a

## Description

We are looking for a Principal Production Engineer to join our team at Zscaler. This role is available as a hybrid opportunity 3 days a week in San Jose, CA or Remote reporting to Production Engineering in the Cloud Infrastructure & Operations department.

In this role, you will provide the technical vision and hands-on execution to drive an "automation-first" culture across the company. By maturing our observability and architectural standards, you will directly reduce our Mean Time to Mitigate (MTTM) and shape the scalability of our globally distributed, multi-cloud infrastructure.

Responsibilities:

- Design and implement highly available, scalable infrastructure across AWS, GCP, and bare-metal environments

- Drive an "automation-first" culture by writing code (Python/Go) to eliminate manual toil and build self-healing systems

- Implement and maintain sophisticated observability (Prometheus, Grafana, OpenTelemetry), define SLIs/SLOs, and establish error budgets

- Act as a lead Incident Commander (TDO on-call), develop response playbooks, and conduct deep-dive post-incident analyses

- Partner with Engineering and partner teams to conduct operability reviews

Requirements:

- Foundational understanding of AI/ML technologies and experience leveraging, securing, or positioning AI-driven solutions to optimize outcomes within your functional domain

- 10+ years of experience managing reliability, scalability, and availability for large-scale production services

- Deep expertise in programming (e.g., Python, Go, or C/C++)

- Strong background in networking protocols, Linux/RHEL systems, and distributed architecture

- Experience in high-stakes incident management and participation in a 24/7 on-call rotation

- Proficiency in leveraging ITIL frameworks and incident data to drive service maturity through systematic problem management and technical operability reviews

Benefits:

- Various health plans

- Time off plans for vacation and sick time

- Parental leave options

- Retirement options

- Education reimbursement

- In-office perks, and more!

## Skills

### Required
- AI/ML
- Python
- Go
- C/C++
- Linux/RHEL
- distributed architecture
- networking protocols
- incident management
- ITIL frameworks

### Nice to have
- AI technologies
- public cloud (AWS, Azure, GCP)
- Infrastructure-as-Code (Ansible, Terraform, Helm, Temporal)
- chaos engineering
- disaster recovery planning
- global routing (BGP)
- traffic tunneling (GRE, IPSec)
- L7 proxy architectures (HAProxy)
- DNS at scale
- OS networking stack internals

---

Source: [Apply at job-boards.greenhouse.io](https://job-boards.greenhouse.io/zscaler/jobs/5149607007?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)