Senior Software Engineer

NVIDIA's DGX Cloud team is seeking a Senior Software Engineer for the Recovery and Resilience team. You will own AI infrastructure services, tools, and open-source projects to support resilient and efficient training from design through production. You will also mentor engineers and lead the team's operations.

Key Responsibilities:

• Design, build, launch, and operate services, tools, and open-source projects
• Plan and assign team work, mentor junior engineers, and review designs
• Build resilient AI infrastructure on Kubernetes and Slurm for NVIDIA's training workloads and customer workloads
• Develop APIs connecting to NVIDIA's resiliency systems, including telemetry, break/fix automation, and checkpoint and execution systems
• Create a job submission layer that hides infrastructure failures from users and restarts long-running training jobs automatically
• Build agentic tools to help customers improve training efficiency by finding bottlenecks, diagnosing failures, and recommending fixes
• Drive open-source work, releasing components for customers and the community to use and improve
• Set up team services, including support, monitoring, alerts, CI/CD, integration test gates, and on-call plans

Requirements:

• BS, MS, or PhD in Computer Science or a related field, plus 12+ years of experience
• Proven experience taking large systems from design through production
• Strong experience with Kubernetes and/or Slurm at scale
• Hands-on experience with PyTorch, TensorFlow, JAX, or Ray
• Strong understanding of NVIDIA GPUs and networking, including failure modes
• Experience running production services, including CI/CD, monitoring, alerting, and on-call

Nice to Have:

• Hands-on experience with fault tolerance or checkpointing for large-scale distributed training on GPU supercomputers
• Demonstrated experience building LLM-based agents or AI tools for performance analysis or troubleshooting
• Background in building external services or maintaining open-source projects in ML infrastructure

Compensation: The expected base salary ranges for this position are: 224,000 USD – 356,500 USD for Level 5; and 272,000 USD – 431,250 USD for Level 6. You will also be eligible for equity and benefits.