Description
NVIDIA is looking for a Software Engineer, Performance Instrumentation to join its team in Santa Clara, CA. This role involves working on groundbreaking projects, driving innovation at the intersection of AI and accelerated computing, and having a direct impact on the future of technology.
Responsibilities
- Define and maintain performance data requirements from various sources, including profiling libraries, annotation APIs, frameworks, runtimes, communication libraries, schedulers, configurations, and infrastructure telemetry.
- Identify and prove visibility gaps with concrete evidence, demonstrating which diagnoses are not possible with current data. Collaborate with relevant teams to address these gaps or build solutions where necessary.
- Inform the build of low-overhead instrumentation and annotation APIs from the perspective of the consumer, ensuring the necessary information is conveyed efficiently and effectively.
- Ensure data quality at ingestion by implementing validation, sanitization, checks, and alerting mechanisms, so that any gaps or errors are detected before impacting users or automated analysis.
- Measure and manage the overhead and observer effects of instrumentation on real workloads, ensuring they remain within acceptable limits.
- Collaborate with multiple partner teams simultaneously, managing commitments and ensuring continuous progress.
Requirements
- Bachelor’s degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
- 8+ years of relevant software engineering experience, including hands-on experience with instrumentation or tracing in production machine learning or systems software.
- Proficiency in CUDA, and working knowledge of machine learning frameworks like PyTorch or JAX, as well as annotation or tracing APIs such as NVTX.
- Demonstrated experience in maintaining data quality in a pipeline, including validation, checks, alerting, and provenance.
- Proven ability to measure instrumentation overhead and make trade-offs among overhead, fidelity, sampling, and aggregation.
- Proven history of implementing modifications in code managed by other groups by identifying requirements, reaching consensus on interfaces, and ensuring completion through to delivery.
- Strong debugging skills across component boundaries and the ability to validate changes through code review, tests, and independent measurements, including using automated development tools.
- Excellent communication skills, especially in conveying evidence, uncertainty, and next steps with engineers and data users.
- Proficiency in Python and C++.
Nice to Have
- Experience establishing telemetry conventions or schemas adopted by other teams.
- Background in storage, checkpointing, or data-loading performance.
- Expertise in crafting low-overhead event collection.
- Familiarity with network and collective-communication telemetry, including NCCL, NICs, and fabric counters.
The base salary range is 168,000 USD - 270,250 USD. You will also be eligible for equity and benefits.