# Senior Resiliency Architect

**Company**: NVIDIA
**Location**: Santa Clara, CA
**Experience**: senior
**Job type**: full-time
**Salary**: 184,000 USD – 287,500 USD
**Category**: Engineering
**Industry**: Technology

**Apply**: https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCareerSite/job/US-CA-Santa-Clara/Senior-Resiliency-Architect--Hardware-Diagnostics-and-Resiliency-Modeling_JR2029227?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_01794029-01e

## Description

We are seeking a Senior Resiliency Architect to support the development of hardware diagnostics and resiliency models for Datacenter and Edge Computing products with applications spanning Agentic and Physical AI.

In this role, you will be a key member of a team of innovators, challenging the status quo and pushing beyond boundaries. You will have the opportunity to impact the industry's leading GPUs and SoCs powering product lines ranging from the rapidly growing field of artificial intelligence to self-driving cars and robots.

**Responsibilities**

- Develop and maintain diagnostics software suite to efficiently stress test NVIDIA GPUs and SOCs to identify hardware defects, including defects that cause silent data corruption. These tests will run in large-scale deployments of Datacenter GPUs and Safety SOCs in package/board/rack configurations spanning GPUs, CPUs, and Networking SOCs.

- Analyze coverage gaps in NVIDIA diagnostic suite flagged by silicon failures on customer workloads or test suites. Enhance diagnostics to improve coverage, repeatability of failures detected, and optimize test time.

- Model and analyze RAS metrics like Failures in Time (FIT) for permanent and transient errors, and Availability from GPU to Rack to Datacenter. Use models to identify gaps and drive RAS improvements through new hardware and software features.

- Analyze field data on hardware interruptions and permanent failures, enhance RAS models to better correlate to field data. Ensure optimal fault attribution, containment, and recovery for hardware errors.

- Design, build, and maintain automation scripts and tooling to systematically kick off, monitor, and manage diagnostic jobs across multiple independent compute clusters.

- Develop and maintain dashboards that integrate diverse data sources and formats, deliver insights through analytics and visualization, and ensure reliable hosting, secure access, and multi-user performance.

- Collaborate with engineering teams to ensure system telemetries accurately categorize cluster failures into hardware faults, test-script bugs, or environment configuration issues.

- Ensure availability models and hardware diagnostics evolve to reflect AI workload changes

**Requirements**

- Master's degree in Computer Engineering, Electrical Engineering, Computer Science or closely related degree or equivalent experience.

- At least 5+ years of relevant experience.

- Knowledge of GPU Architecture and programming, hardware diagnostics

- Good understanding of Reliability (FIT) models, AVF estimation, Availability concepts, ECC/parity/CRC strategies.

- Proficiency in data mining, automation, usage of AI tools and scripting with Python, SQL or similar.

- Proficiency in C/C++, CUDA programming, and using AI tools for coding and debug.

- Strong debugging and analytical skills.

- Excellent interpersonal skills and ability to collaborate with on-site and remote teams.

- Working knowledge of hardware diagnostics, and container-based job execution.

**Nice to Have**

- Strong statistical judgment - particularly around exposure-adjusted rates, sample size, and distinguishing signal from artefact.

- Strong understanding of resiliency and reliability trade-offs in AI data centers, including failure modes, mitigation and recovery strategies, and impact on large-scale training/inference workloads

- Experience in developing embedded software or visualization and data analytics tools.

The base salary will be determined based on your location, experience, and other objective job-related factors. The expected base salary range for this position is 184,000 USD – 287,500 USD. You will also be eligible for equity and benefits.

## Skills

### Required
- GPU Architecture
- hardware diagnostics
- Reliability (FIT) models
- AVF estimation
- Availability concepts
- ECC/parity/CRC strategies
- Python
- SQL
- C/C++
- CUDA programming

### Nice to have
- statistical judgment
- resiliency and reliability trade-offs in AI data centers
- embedded software
- visualization and data analytics tools

---

Source: [Apply at nvidia.wd5.myworkdayjobs.com](https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCareerSite/job/US-CA-Santa-Clara/Senior-Resiliency-Architect--Hardware-Diagnostics-and-Resiliency-Modeling_JR2029227?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)