Description
Grafana Labs is seeking a Staff Software Engineer to focus on database reliability engineering for Grafana Cloud. The successful candidate will work on building and operating large-scale multi-tenant SaaS databases, including Mimir, Loki, Tempo, and Pyroscope.
About the role:
The role combines building and operating. You’ll write the software, automation, and product changes that make our databases more reliable, including changes to the Mimir, Loki, and Tempo codebases where reliability calls for it. You’ll also be on call for our highest-SLA customers and handle the operational work that comes with it.
Responsibilities:
- Build reliability into the product by writing production code in our services, tooling, and the Mimir, Loki, and Tempo codebases.
- Develop automation and platform tooling to remove toil and scale reliability practices.
- Lead technical design, author design documents, and review others’ designs and PRs.
- Own the SLO framework, defining and evolving per-tenant SLOs and reliability models.
- Improve observability and alerting to find problems before customers do.
- Be on call for our highest-SLA customers and handle incident response.
- Multiply the team by mentoring engineers and teaching reliability practices.
Requirements:
- 8+ years of software engineering experience, including significant time building and owning distributed systems in production.
- Strong production software skills, ideally in at least one object-oriented language.
- Deep understanding of distributed systems, performance, scaling, and failure modes.
- Experience running multi-tenant systems at scale on Kubernetes in AWS, GCP, or Azure.
- Hands-on experience designing and implementing SLOs.
- Experience being on call for production systems.
- Staff-level technical leadership.
Nice to Have:
- Experience with Mimir, Loki, Tempo, Pyroscope, Prometheus, or other observability and time-series databases.
- Open-source contributions.
- Customer reliability engineering (CRE) or customer-facing escalation experience.
On-Call:
Weekday on-call is 12-hour shifts, Monday to Friday, split between North America and Europe. Weekends are covered by a separate 24-hour rotation.
Compensation and Benefits:
The base compensation range for this role in the UK is £103,958 - £124,750. Benefits include equity, bonus (if applicable), and other benefits.