# AI Reliability Engineer, HPC

**Company**: Microsoft AI
**Location**: London
**Work arrangement**: hybrid
**Experience**: senior
**Job type**: full-time
**Salary**: £93,500.00 - £161,800.00 per year
**Category**: Engineering
**Industry**: Technology

**Apply**: https://microsoft.ai/job/ai-reliability-engineer-hpc/?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_d2fad152-388

## Description

We're seeking an experienced AI Reliability Engineer to join our High Performance Computing (HPC) infrastructure team. You'll blend software engineering and systems engineering to keep our large-scale distributed AI infrastructure reliable and efficient.

**Responsibilities**

- Ensure uptime, resiliency, and fault tolerance of HPC clusters powering MAI model training and inference.

- Design and maintain monitoring, alerting, and logging systems to provide real-time visibility into HPC systems.

- Build automation for deployments, incident response, scaling, and failover in CPU+GPU environments.

- Lead on-call rotations, troubleshoot production issues, and drive continuous improvements.

- Ensure data privacy, compliance, and secure operations across model training and serving environments.

- Partner with ML engineers and platform teams to improve developer experience and accelerate research-to-production workflows.

**Qualifications**

- Bachelor's Degree in Computer Science or related technical discipline AND 4+ years technical experience in Site Reliability Engineering, DevOps, or Infrastructure Engineering.

- Experience with Kubernetes, Docker, container orchestration, and CI/CD pipelines for ML training or inference workloads.

- Experience with public cloud platforms such as Azure, AWS, or GCP, including infrastructure-as-code.

- Experience with monitoring and observability tools such as Grafana, Datadog, or OpenTelemetry.

- Programming or scripting experience in Python, Go, or Bash.

- Experience with distributed systems, networking, storage, and high-performance computing (HPC).

Software Engineering IC5 - The typical base pay range for this role across United Kingdom is £93,500.00 - £161,800.00 per year.

## Skills

### Required
- Kubernetes
- Docker
- container orchestration
- CI/CD pipelines
- public cloud platforms
- infrastructure-as-code
- monitoring and observability tools
- Python
- Go
- Bash
- distributed systems
- networking
- storage
- high-performance computing (HPC)

### Nice to have
- Master's Degree in Computer Science or related technical discipline
- Experience with GPU clusters and workload schedulers for ML/AI workloads
- Experience with ML training or inference pipelines
- Experience with capacity planning and cost optimization for GPU-based infrastructure

---

Source: [Apply at microsoft.ai](https://microsoft.ai/job/ai-reliability-engineer-hpc/?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)