Description
We're looking for a deeply technical Senior HPC Cluster Administrator to lead the design, deployment, and reliability of our large-scale GPU compute clusters. These systems run the most demanding deep learning training, inference, and high-performance computing workloads in the industry , from DGX/HGX platforms to ground-breaking Grace Blackwell systems.
The ideal candidate will have a strong background in Linux systems administration, experience with large-scale HPC or ML training clusters, and proficiency with scripting languages such as Python and/or bash. They will also have experience with Slurm, configuration management, and IaC tools like Ansible and Terraform.
Responsibilities:
- Own the full lifecycle of GPU compute clusters , procurement, provisioning, configuration management, monitoring, and deprecation , across heterogeneous Linux environments (DGX, HGX, embedded systems)
- Design and scale storage solutions (NFS, Lustre, WekaFS, or equivalent) with a clear roadmap for capacity and performance growth
- Lead automation of infrastructure using modern IaC tools (Ansible, Terraform) and CI/CD pipelines (GitLab)
- Manage and optimize job scheduling via Slurm, including fair-share policies, reservation management, and MIG/GPU partitioning strategies
- Maintain and improve observability stacks (Prometheus, Grafana, DCGM) and drive proactive resolution of hardware and software incidents
- Collaborate with ML engineers and software teams to tune cluster configuration for large-scale distributed training workloads
- Evaluate and introduce new technologies , networking fabrics (InfiniBand, NVLink, EFA/RDMA), storage tiers, container runtimes , to improve performance and reliability
- Mentor junior engineers and contribute to team-wide engineering standards
Requirements:
- BS/MS in CS, EE, CE, or equivalent hands-on experience
- 5+ years of experience deploying and administering large-scale HPC or ML training clusters
- Deep expertise in Linux systems administration at scale
- Strong scripting and automation skills in Python and/or bash
- Hands-on experience with Slurm (scheduling, accounting, cgroup configuration)
- Proficiency with configuration management and IaC (Ansible required; Terraform a plus)
- Experience with container technologies (Docker, Apptainer/Singularity, Kubernetes)
- Solid understanding of high-speed networking (InfiniBand, RoCE, RDMA, EFA)
- Experience with distributed/parallel filesystems and storage architecture
- Ability to own problems end-to-end and communicate clearly with engineering and management stakeholders
Preferred qualifications include experience with NVIDIA GPU infrastructure tools, familiarity with cluster management platforms, and knowledge of MLOps tooling or ML platform engineering.