# Software Engineer, Collective Communication and Network Performance

**Company**: NVIDIA
**Location**: Santa Clara, CA
**Experience**: senior
**Job type**: full-time
**Salary**: 172,500 USD – 338,000 USD
**Category**: Engineering
**Industry**: Technology

**Apply**: https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCareerSite/job/US-CA-Santa-Clara/Software-Engineer--Collective-Communication-and-Network-Performance_JR2030158?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/job/software-engineer-collective-communication-and-24a0a873-4af/

## Description

NVIDIA is looking for a Software Engineer to join their team in Santa Clara, CA. The role involves developing software tools to detect and diagnose collective communication and network performance problems in AI workloads across thousands of GPUs.

**Job Description:** As a Software Engineer, you will develop detection algorithms to identify communication performance problems from always-on telemetry. You will distinguish the causes of problems, such as late rank, degraded GPU, fabric congestion, silent NIC fault, slower path chosen over NVLink, or hang.

**Key Responsibilities:**

- Develop detection algorithms to identify communication performance problems

- Build the analysis infrastructure for these algorithms

- Define the data needed from NCCL profiler and tuner plugin interfaces, RDMA transport state, and NIC and switch counters

- Reproduce communication problems on internal clusters and verify automated diagnoses

- Extend coverage beyond NCCL to point-to-point and other communication paths

**Requirements:**

- Bachelor's degree in Computer Science, Computer Engineering, or a related field

- 8+ years of relevant software engineering experience

- Working knowledge of NCCL or a comparable library

- Working knowledge of RDMA networking in practice

- Demonstrated experience diagnosing multi-node communication problems

- Proven history of landing changes in code owned by other teams

- Strong debugging skills across component boundaries

- Proficiency in C++ and Python

**Nice to Have:**

- Contributions to NCCL, UCX, libfabric, NVSHMEM, or a comparable communication library

- Experience with point-to-point and expert-parallel communication paths

- Experience building detection or root-cause analysis that ran in production

- Experience operating or debugging training or inference jobs above a thousand GPUs

**Salary:** The expected base salary range for this position is 172,500 USD – 338,000 USD.

For more fresh new jobs, check out [yubhub.co](https://yubhub.co/).

## Skills

### Required
- C++
- Python
- NCCL
- RDMA
- network performance

### Nice to have
- point-to-point communication paths
- expert-parallel communication paths
- detection or root-cause analysis
- large-scale GPU computing

---

Source: [Apply at nvidia.wd5.myworkdayjobs.com](https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCareerSite/job/US-CA-Santa-Clara/Software-Engineer--Collective-Communication-and-Network-Performance_JR2030158?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)
