# Principal Software Engineer, At-Scale Reliability and Fleet Intelligence — CSP Engagements

**Company**: NVIDIA
**Location**: Santa Clara, CA
**Experience**: senior
**Job type**: full-time
**Category**: Engineering
**Industry**: Technology

**Apply**: https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCareerSite/job/US-CA-Santa-Clara/Principal-Software-Engineer--At-Scale-Reliability-and-Fleet-Intelligence---CSP-Engagements_JR2020320?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_fd543d27-f2e

## Description

We are seeking a Principal Software Engineer to join our CSP Engagements team, focusing on fleet-scale reliability. You will work directly with key CSP/hyperscale customers' engineering teams to ensure NVIDIA platforms achieve target MTBI in production.

**Responsibilities:**

- Drive reliability work streams with CSP engineering teams, ensuring shared understanding of MTBI measurement methodology, failure classification, and health monitoring architecture.

- Gather and synthesize CSP fleet reliability data, identifying failure patterns and championing improvements back into NVIDIA's firmware, driver, and hardware teams.

- Define a consistent MTBI measurement methodology that works across different CSP monitoring environments and operational practices.

- Conduct fleet-scale failure pattern analysis using statistical methods to classify failures as systemic, environmental, or configuration-specific.

- Drive fleet health monitoring integration architecture, ensuring NVIDIA's health agents, telemetry, and reporting align with CSP operational workflows and automation.

- Define burn-in reliability test environment and cluster certification criteria in collaboration with quality teams.

- Collaborate with CSPs to ensure reliability-related integration work is complete ahead of at-scale launch.

- Develop predictive failure models using fleet telemetry and validate their effectiveness in customer environments.

**Requirements:**

- 15+ years of experience in systems software at datacenter scale or reliability engineering with focus on at-scale challenges.

- BS or MS in Computer Science, Electrical Engineering, Statistics, or related field.

- Deep expertise in multi-NUMA, rack-scale system software and firmware, and statistical failure analysis methods.

- Experience with fleet-level telemetry and observability systems.

- Understanding of hardware failure modes in large-scale GPU/accelerator deployments.

- Experience defining or operating burn-in, stress testing, or certification frameworks for complex hardware systems.

- Customer obsession and strong communication skills.

**Benefits:**

- Equity eligibility

- Benefits package

This is a full-time position, and applications will be accepted until June 30, 2026.

## Skills

### Required
- multi-NUMA system software
- rack-scale system firmware
- statistical failure analysis
- fleet-level telemetry
- predictive failure modeling
- hardware failure modes
- burn-in testing
- certification frameworks

### Nice to have
- hyperscaler experience
- NVIDIA GPU error taxonomy
- health scoring
- anomaly detection
- MTBI/MTBF measurement standards

---

Source: [Apply at nvidia.wd5.myworkdayjobs.com](https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCareerSite/job/US-CA-Santa-Clara/Principal-Software-Engineer--At-Scale-Reliability-and-Fleet-Intelligence---CSP-Engagements_JR2020320?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)