Description
We are seeking an experienced Senior AIOps Engineer to design, build, and operate the autonomous agents that turn observability data into safe, closed-loop operational action.
You will sit at the intersection of service observability and agentic AI: connecting agents to metrics, logs, traces, events, change data, and service topology so they can detect, diagnose, and remediate production issues with minimal human intervention.
In this role, you will:
- Design and build autonomous AIOps agents (LLM-based and multi-agent) that run the full operational loop , observe, correlate, diagnose, decide, act, and verify , across incident triage, root-cause analysis, and auto-remediation
- Connect agents to the observability estate (OpenTelemetry, Grafana, Tempo/Jaeger, APM, CMDB, change and deployment events) through well-defined tools/APIs and MCP servers, giving agents real-time, topology-aware context to reason over
- Turn observability signals into autonomous action: build event-driven triggers on SLO burn-rate, anomaly, and correlated alert patterns that launch agent workflows such as rollback, restart, scale-out, failover, traffic shift, config fix, or ticket enrichment
- Build the agent action layer: convert runbooks into machine-executable, idempotent actions (runbook-as-code), with pre-checks, blast-radius limits, dry-run modes, automatic rollback, and post-action health verification against SLIs
- Implement guardrails and human-in-the-loop controls: risk-tiered autonomy levels, policy-as-code, approval gates for high-risk changes, full audit trails, and explainable reasoning for every decision an agent makes
- Apply AI to observability data: anomaly detection, forecasting, alert noise reduction, event correlation, and retrieval-augmented reasoning over past incidents, runbooks, and post-mortems to speed up RCA and cut MTTD/MTTR
- Build evaluation and observability for the agents themselves: trace agent reasoning and tool calls, measure accuracy, action success rate, false-action rate, latency, and cost; run replay/simulation and chaos tests before increasing autonomy in production
- Partner with SRE, platform, and application teams to onboard services to autonomous operations, set standards and reusable patterns, and coach engineers on using AI + observability to automate their own operational workflows
To be successful in this role, you should meet the following requirements:
- 10+ years overall experience with 5+ years in SRE, AIOps, observability, or platform engineering.
- Proven hands-on development of autonomous agents using frameworks such as LangGraph, LangChain, CrewAI, AutoGen, Semantic Kernel, or equivalent; strong grasp of tool/function calling, MCP, planning, memory, and multi-agent orchestration
- Deep service observability expertise: OpenTelemetry, Grafana, Jaeger/Tempo, or enterprise APM (Splunk, AppDynamics); strong PromQL/LogQL/SQL and SLO/SLI/error-budget design
- Track record of delivering closed-loop auto-remediation in production that measurably reduced MTTR, alert noise, or manual toil, with clear safety controls and rollback
- Strong Python (Go a plus) with experience building event-driven services and integrations across Kubernetes, CI/CD, ITSM (ServiceNow), and automation tools.
- Practical knowledge of applied ML and LLM techniques for operations: anomaly detection, time-series forecasting, event correlation, RAG, prompt engineering, and evaluation of LLM outputs
- Understanding of responsible and secure AI in regulated environments: least-privilege agent access, PII redaction, auditability, model risk management, and change-control compliance
- Demonstrated technical leadership, with experience mentoring and leading engineers, setting engineering direction, reviewing designs, and developing team capability while influencing risk partners and senior stakeholders