Description
The Collaboration Technology Group at Cisco is seeking a Software Engineer to design, build, and deploy the core capabilities of their next-generation AI-powered Production Intelligence platform.
You will combine Site Reliability Engineering practices with modern agentic AI to transform how engineering and leadership teams monitor, diagnose, and auto-remediate global SaaS infrastructure.
Responsibilities:
- Technical Design & Architecture: Design and implement high-resilience software systems for AI-assisted observability, automated incident response, and self-healing cloud infrastructure.
- Agentic Workflows & Tooling: Design, build, and maintain production-grade AI agents, MCP tool integrations, and deterministic evaluation pipelines for automated operational decision support.
- Telemetry & Insights: Implement ingestion and correlation pipelines across distributed logs, metrics, OpenTelemetry traces, change events, and runbooks to accelerate Mean Time to Detection (MTTD) and Resolution (MTTR).
- Safe Production Automation: Develop proactive anomaly detection and Human-in-the-Loop (HITL) remediation workflows with rigorous safety, security, and quality guardrails.
- Reliability & Scalability Engineering: Partner with application and infrastructure teams to define SLIs/SLOs, handle error budgets, and lead deep-dive post-incident reviews (PIRs).
- Mentorship & Best Practices: Mentor mid-level and junior engineers, conduct thorough code reviews, establish engineering guidelines, and drive operational excellence across global development and operations teams.
- Cross-Team Delivery: Manage priorities and deadlines, communicate progress clearly, and work across teams to turn production needs into reliable software and AI-assisted capabilities.
Minimum Qualifications:
- Bachelor’s degree + 8 years of related experience, Master’s + 6 years, or PhD + 3 years in Computer Science, Software Engineering, or a related technical field.
- Experience as a Senior / Lead SRE or Software Engineer delivering distributed, high-availability SaaS platforms at scale.
- Strong proficiency in Python, Go, Java, or C++ with experience designing microservices, APIs, and production automation.
- Deep experience with Kubernetes, Docker, and container orchestration in large-scale multi-cluster environments.
- Proven background in SRE practices: SLI/SLO design, observability platforms (metrics/logs/traces), incident management, and automated root cause analysis (RCA).
Preferred Qualifications:
- Experience building LLM pipelines, AI Agents, Model Context Protocol (MCP) servers/clients, RAG architectures, and evaluation frameworks.
- Experience with OpenTelemetry (OTel), Prometheus, Grafana, Splunk, ThousandEyes, or distributed tracing systems.
- Expertise in public cloud providers (AWS, GCP, Azure), Terraform/IaC, and GitOps/CI/CD pipelines (Jenkins, GitHub Actions, Harness).
- Experience implementing responsible AI guardrails, deterministic fallback logic, and policy-driven remediation engines.
- Experience with streaming and data platforms (Kafka, Redis, PostgreSQL, Elasticsearch/Vector DBs).