Senior Site Reliability Engineer - Observability

Job Overview

We are seeking a highly technical Senior Observability Site Reliability Engineer with a specialty in Splunk to own and evolve our Splunk ecosystem. In this role, you will move beyond simple monitoring to delivering a world-class, comprehensive, scalable Observability Platform that enables our SRE teams and business partners.

Key Responsibilities

• Design, build, and maintain scalable observability infrastructure using tools like Terraform.
• Optimize the collection, processing, and storage of log data to ensure high reliability and low latency of our Splunk services.
• Participate in on-call rotations and lead post-incident reviews to drive systemic improvements and 'observability-driven development.'
• Eliminate 'toil' by automating the deployment and scaling of observability agents and collectors.

Required Skills & Experience

• Minimum 5+ years of experience scaling and managing Splunk Cloud at scale (1000+ SVCs), including Workload Management (WLM) and HEC optimization.
• Expertise in creating intuitive, actionable Splunk dashboards that correlate data across multiple sources.
• Minimum 3+ years of experience in an SRE, DevOps, or Systems Engineering role with a focus on high-availability systems.
• Strong coding skills in SPL, Go, Python, or Ruby for building internal tools and automating workflows.
• Deep understanding of Linux internals, networking (TCP/IP, DNS, Load Balancing), and container orchestration (Kubernetes/EKS).
• A data-driven approach to debugging complex, cross-service performance bottlenecks.

Bonus Skills

• Hands-on experience with OpenTelemetry (OTel), Vector, or similar frameworks for instrumenting applications.
• Experience in implementing Splunk charge-back app for usage reporting.
• Experience managing observability native tools within AWS or GCP.

Additional Requirements

• This position requires the ability to access federal environments and/or have access to protected federal data. As a condition of employment for this position, the successful candidate must be able to submit documentation establishing U.S. Person status.
• This person must attend in-person onboarding in our San Francisco office the first week of employment.

Salary Information

The annual base salary range for this position for candidates located in California (excluding San Francisco Bay Area), Colorado, Illinois, New York, and Washington is between: $147,000-$202,000 USD