Description
The Splunk Agent Resilience team is defining the future of AI resilience. Our team provides scalable, cost-effective evaluation and guardrails that ensure AI agents behave as intended, improving reliability and reducing risks.
As a Staff Site Reliability Engineer (SRE), you will provide technical leadership for the reliability, scalability, and operational architecture of Splunk Agent Resilience's platform. You will define the long-term reliability strategy, lead major infrastructure initiatives, and drive engineering excellence across deployment automation, production operations, and platform resiliency.
Your Impact
- Define and drive the technical roadmap for platform reliability, scalability, and operational excellence.
- Lead the architecture and evolution of deployment platforms supporting cloud and air-gapped customer environments.
- Establish reliability engineering standards, including service level objectives (SLOs), operational readiness, capacity planning, and resiliency reviews.
- Lead major reliability and scalability initiatives across Kubernetes, deployment infrastructure, databases, and networking.
- Drive automation to eliminate operational toil and improve engineering productivity.
- Design and build internal platforms, frameworks, and tooling that enable reliable operations at scale.
- Lead complex production incident response and drive systemic improvements through root cause analysis and long-term remediation.
- Partner with engineering leadership to influence platform architecture, deployment strategy, and production readiness.
- Mentor engineers and raise the engineering bar through technical leadership, design reviews, and operational guidelines.
- Collaborate with customers and internal teams to build secure, scalable, and highly reliable deployment architectures for both cloud and on-prem environments.
Minimum Qualifications
- 10+ years of experience in Site Reliability Engineering, DevOps, Platform Engineering, Infrastructure Engineering, or a related field
- 5+ years' operating large-scale Kubernetes platforms in production.
- Experience designing highly available, scalable, and resilient distributed systems.
- Experience with AWS, GCP, or other public cloud platforms.
- Strong experience designing CI/CD platforms and deployment automation at scale.
Preferred Qualifications
- Expertise in observability, monitoring, alerting, capacity planning, and performance engineering. Strong programming skills in Python and/or Go.
- Deep experience with Infrastructure as Code (Terraform or similar)
- Experience operating and tuning data infrastructure at scale , OLTP/OLAP databases, message queues, and object storage systems
- Experience with MLOps preferred
- Strong understanding of networking, distributed systems, storage, databases, and cloud architecture
- Experience operating both SaaS and enterprise/on-prem deployments
- Experience leading incident management, postmortems, and long-term reliability initiatives
- Ability to collaborate effectively across engineering, infrastructure, security, and customer-facing teams