Principal Production Engineer

We are looking for a Principal Production Engineer to join our team at Zscaler. This role is available as a hybrid opportunity 3 days a week in San Jose, CA or Remote reporting to Production Engineering in the Cloud Infrastructure & Operations department.

In this role, you will provide the technical vision and hands-on execution to drive an "automation-first" culture across the company. By maturing our observability and architectural standards, you will directly reduce our Mean Time to Mitigate (MTTM) and shape the scalability of our globally distributed, multi-cloud infrastructure.

Responsibilities:

• Design and implement highly available, scalable infrastructure across AWS, GCP, and bare-metal environments
• Drive an "automation-first" culture by writing code (Python/Go) to eliminate manual toil and build self-healing systems
• Implement and maintain sophisticated observability (Prometheus, Grafana, OpenTelemetry), define SLIs/SLOs, and establish error budgets
• Act as a lead Incident Commander (TDO on-call), develop response playbooks, and conduct deep-dive post-incident analyses
• Partner with Engineering and partner teams to conduct operability reviews

Requirements:

• Foundational understanding of AI/ML technologies and experience leveraging, securing, or positioning AI-driven solutions to optimize outcomes within your functional domain
• 10+ years of experience managing reliability, scalability, and availability for large-scale production services
• Deep expertise in programming (e.g., Python, Go, or C/C++)
• Strong background in networking protocols, Linux/RHEL systems, and distributed architecture
• Experience in high-stakes incident management and participation in a 24/7 on-call rotation
• Proficiency in leveraging ITIL frameworks and incident data to drive service maturity through systematic problem management and technical operability reviews

Benefits:

• Various health plans
• Time off plans for vacation and sick time
• Parental leave options
• Retirement options
• Education reimbursement
• In-office perks, and more!