Description
Mastercard's Operational Intelligence team is building the next generation of data products and AI solutions that help customers access and use operational insights in new ways.
As a Senior AI Product Ops AI Ops, you will own end-to-end production monitoring for one or more AI agents, tracking key metrics, KPIs, accuracy, latency, SLA breaches, guardrail triggers, and out-of-scope rates through dedicated observability dashboards.
Your responsibilities will include:
- Serving as L1/L2 production support, providing first response monitoring, triage, and escalation to tech teams as new agentic services go live
- Identifying and diagnosing issues by monitoring eval scores and drift alerts, inspecting failing traces and patterns, and classifying root causes such as intent, tool, parameter, or hallucination errors
- Evaluating whether the agent's reasoning remains correct in production, checking whether context retrieval stays current, whether resolutions match ground truth, and whether the learning loop is drifting
- Monitoring and managing post-production reliability, including API and authentication failures and issues originating from external AI endpoints, to protect revenue-generating, client-facing agents
- Partnering with AI Engineering to prioritize fixes, run experiments on prompts, context, and routing in Dev, and validate improvements via evals before handing off proven changes for deployment
- Embedding governance within the agent's learning loop so that approval workflows and audit logging travel with every production change, rather than being added after the fact
To be successful in this role, you will need:
- Experience monitoring or operating production AI/ML or agentic systems, including observability, evaluation, and drift detection practices
- Strong analytical skills to diagnose root causes across intent classification, tool selection, parameter errors, and hallucination patterns
- Understanding of AI/LLM observability concepts such as tracing, telemetry, guardrails, and evaluation engineering, including benchmarks, automated evals, human review, and regression testing
- Familiarity with governance, compliance, and responsible AI practices such as PII detection, access control, audit logging, and policy enforcement is a plus
- Excellent communication skills to translate production signals into clear, prioritized asks for AI Product and AI Engineering teams
- Comfort working in a fast-evolving environment where the process itself, not just the throughput, is what is being continuously operated on and improved