# Senior Software Engineer

**Company**: NVIDIA
**Location**: Santa Clara, CA
**Experience**: senior
**Job type**: full-time
**Salary**: 224,000 USD – 356,500 USD for Level 5; and 272,000 USD – 431,250 USD for Level 6
**Category**: Engineering
**Industry**: Technology

**Apply**: https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCareerSite/job/US-CA-Santa-Clara/Senior-System-Software-Engineer--DGX-Cloud-AI-Infrastructure_JR2029457?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/job/senior-software-engineer-1f052312-956/

## Description

NVIDIA's DGX Cloud team is seeking a Senior Software Engineer for the Recovery and Resilience team. You will own AI infrastructure services, tools, and open-source projects to support resilient and efficient training from design through production. You will also mentor engineers and lead the team's operations.

**Key Responsibilities:**

- Design, build, launch, and operate services, tools, and open-source projects

- Plan and assign team work, mentor junior engineers, and review designs

- Build resilient AI infrastructure on Kubernetes and Slurm for NVIDIA's training workloads and customer workloads

- Develop APIs connecting to NVIDIA's resiliency systems, including telemetry, break/fix automation, and checkpoint and execution systems

- Create a job submission layer that hides infrastructure failures from users and restarts long-running training jobs automatically

- Build agentic tools to help customers improve training efficiency by finding bottlenecks, diagnosing failures, and recommending fixes

- Drive open-source work, releasing components for customers and the community to use and improve

- Set up team services, including support, monitoring, alerts, CI/CD, integration test gates, and on-call plans

**Requirements:**

- BS, MS, or PhD in Computer Science or a related field, plus 12+ years of experience

- Proven experience taking large systems from design through production

- Strong experience with Kubernetes and/or Slurm at scale

- Hands-on experience with PyTorch, TensorFlow, JAX, or Ray

- Strong understanding of NVIDIA GPUs and networking, including failure modes

- Experience running production services, including CI/CD, monitoring, alerting, and on-call

**Nice to Have:**

- Hands-on experience with fault tolerance or checkpointing for large-scale distributed training on GPU supercomputers

- Demonstrated experience building LLM-based agents or AI tools for performance analysis or troubleshooting

- Background in building external services or maintaining open-source projects in ML infrastructure

**Compensation:** The expected base salary ranges for this position are: 224,000 USD – 356,500 USD for Level 5; and 272,000 USD – 431,250 USD for Level 6. You will also be eligible for equity and benefits.

For more fresh new jobs, check out [yubhub.co](https://yubhub.co/).

## Skills

### Required
- Kubernetes
- Slurm
- PyTorch
- TensorFlow
- JAX
- Ray
- NVIDIA GPUs

### Nice to have
- fault tolerance
- checkpointing
- LLM-based agents
- AI tools for performance analysis

---

Source: [Apply at nvidia.wd5.myworkdayjobs.com](https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCareerSite/job/US-CA-Santa-Clara/Senior-System-Software-Engineer--DGX-Cloud-AI-Infrastructure_JR2029457?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)