Description
Disney Entertainment and ESPN Product & Technology is a global organisation of engineers, product developers, designers, technologists, data scientists, and more – all working to build and advance the technological backbone for Disney's media business globally.
The team marries technology with creativity to build world-class products, enhance storytelling, and drive velocity, innovation, and scalability for their businesses. They are Storytellers and Innovators. Creators and Builders. Entertainers and Engineers.
As a Software Engineer II on the Reliability Insights team, you will design and build data and reporting pipelines that evaluate and prove the real-world performance, precision, and business impact of Disney's operational AI and observability tools. You will bridge backend engineering and data-driven insights, transforming raw telemetry, AI agent interactions, and operational logs into clear metrics around AI reliability, model drift, and incident mitigation.
In this role, you will build automated ground-truth pipelines and model evaluation frameworks that quantify how effectively AI agents and anomaly models perform in production. You will capture end-user feedback, correlate AI signals with incident resolution timelines, and build single-pane dashboards for engineering and leadership.
Responsibilities:
- Design, build, and maintain automated feedback pipelines that capture operational outcomes and correlate them directly back to AI model predictions and agent actions.
- Develop programmatic evaluation frameworks to track model precision, recall, and drift over time, establishing automated threshold alerts to proactively trigger model tuning and prevent alert fatigue.
- Build end-to-end data pipelines linking AI outputs to incident lifecycles, quantifying business impact through metrics such as Service to Incident correlation, notification success rates, and Mean Time to Mitigation (MTTM).
- Design and deploy scalable APIs, metrics layers, and single-pane model health dashboards that present clear AI ROI, operational health, and performance narratives to leadership and technical stakeholders.
- Build mechanisms to capture structured feedback from end-users to continuously feed real-world usage signals back into model improvement cycles.
Basic Qualifications:
- 3+ years of applicable experience in backend and data pipeline engineering, including building scalable APIs and metrics/insight generation platforms.
- Strong experience with data platform engineering and distributed processing tools to process and analyse telemetry data.
- Practical understanding of model evaluation concepts, such as precision, recall, model drift, and ground-truth labelling systems.
- Hands-on experience with modern development practices, including version control, containerisation, and cloud-native deployments.
- Proficiency with AI-assisted development tools to accelerate engineering velocity.
- Strong analytical skills to translate complex system telemetry and operational logs into meaningful reliability metrics and business insights.
- Excellent cross-functional collaboration and communication skills, with a track record of partnering with operational teams to turn raw data into actionable insights.
Preferred Qualifications:
- Experience with observability, telemetry, and monitoring platforms.
- Familiarity with agentic workflows and foundation models and how to evaluate their output accuracy.
- Knowledge of incident management workflows and SRE metrics.
- Prior experience working on data attribution, metrics aggregation, or financial/performance visibility tooling in high-scale environments.
The hiring range for this position in New York, NY is $124,900.00 to $167,400.00 per year.