AI Operations Specialist

We are seeking a thoughtful, analytical, and technically curious AI Operations Specialist to help scale and operationalize Agent AI across Customer Care.

This is not an engineering role and not a traditional operations role. AI Operations Specialists sit at the intersection of AI systems, frontline support expertise, and operational rigor - ensuring our AI agents deliver accurate, high-quality resolutions at scale.

In this role, you will be responsible for defining how Agent AI behaves in real-world scenarios by creating and maintaining AI Operating Procedures (AOPs/SOPs), evaluating AI performance through structured QA, and driving iteration loops with Product, Engineering, and external vendors.

Responsibilities

AI Operating Model & SOP Development (35%)

• Design and maintain detailed AI Operating Procedures (AOPs/SOPs) that define how Agent AI should behave across supported use cases.
• Translate human-agent judgment into structured system logic, mapping actions, edge cases, and expected outcomes.
• Inspect and reference APIs, system actions, and integration flows directly within documentation.
• Continuously update SOPs as product functionality and use cases evolve.

AI Quality Assurance & Performance Improvement (35%)

• Define and execute QA frameworks to evaluate containment, accuracy, and resolution quality across vendor bots, internal bots, and LLM-based systems.
• Identify failure modes, escalation drivers, and judgment gaps.
• Partner with Product, Engineering, and vendors to iterate on performance and improve customer experience.
• Enable and calibrate human QA teams evaluating Agent AI at scale.

AI Analytics & Operational Insights (20%)

• Build and maintain dashboards tracking key AI metrics (e.g., containment rate, accuracy, escalation rate).
• Analyze trends and surface actionable insights to guide prioritization and iteration.
• Use data to inform readiness decisions and highlight performance risks.

Cross-Functional Execution (10%)

• Participate in daily standups with Product and Engineering to align on priorities and iteration loops.
• Provide structured operational feedback to vendors and internal stakeholders.
• Support release readiness discussions and post-launch performance reviews.

Requirements

• A college degree or equivalent qualification.
• Three to five years of relevant experience in operations, QA, product operations, support tooling, or adjacent roles.
• A curious, proactive, and entrepreneurial mindset with strong problem-solving skills grounded in critical thinking and common sense.
• Demonstrated ability to reason through complex systems and translate human judgment into structured processes.
• Comfort working with APIs, system actions, and technical documentation (without being an engineer).
• Strong analytical skills and experience working with performance metrics and dashboards.
• Willingness to roll up sleeves and actively participate in execution-heavy work.
• Excellent interpersonal skills and empathy for customers.
• Strong organizational skills and comfort leading workstreams involving Product, Engineering, QA, and external partners.
• Willingness to travel internationally: This role requires travel up to once per quarter to BPO sites supporting AI QA operations.