# Supercompute Operations Engineering

**Company**: SpaceX
**Location**: Southaven, MS
**Work arrangement**: onsite
**Experience**: senior
**Job type**: full-time
**Category**: Engineering
**Industry**: Technology

**Apply**: https://job-boards.greenhouse.io/xai/jobs/5273689007?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_1a600285-3c9

## Description

SpaceXSI is building Super Intelligence to help humanity understand the universe.

The team owns 100% GPU uptime, end to end, 24/7, across both compute and facility. It owns incident response and the engineering behind it, holds central control authority from the customer back to the facility, runs day-to-day sustainment of live datahalls, and leads deployment engineering for new datahall bring-up. The team will also build the automation for continuous response improvement.

## Responsibilities:

- Own monitoring architecture, alerting, signal quality, and automation of monitoring across GPU and host telemetry (DCGM, Prometheus/Grafana) and facility telemetry (BMS, EPMS, DCIM, SCADA, CDU / direct liquid cooling).

- Own compute hardware change incorporation - physical, firmware, software.

- Be part of 24/7 incident response, mitigation, and control authority - end to end - from customer impact back to the facility.

- Own day-to-day site sustainment operations, including resolving issues impacting rack-level compute and facility infrastructure.

- Disposition daily issues across both Site Operations (rack compute) and Facility Operations (power and cooling).

- Lead acceptance and deployment testing campaigns to bring new data halls fully online.

- Develop production automation for new datahall bring-up, automation behind alerting and automated incident response.

- Develop and maintain standard operating procedures and planning for facility and site technicians.

- Own and document the live, changing state of datahall operation as new halls come online.

- Enable Site Operations and Facility Operations technicians to do their day to day jobs more efficiently and safely to increase GPU uptime.

- Create predictive and responsive automation to alerts and incidents to enable your response to be faster and more impactful.

- Lead cross-functional reliability projects spanning compute, network (InfiniBand/RoCE), storage, and facility signal boundaries.

- Truly root cause systemic and design failure classes across hardware and integrated facility and operations systems.

## Basic Qualifications:

- Bachelor's degree in Mechanical, Electrical, or Computer Engineering or a related scientific discipline and 2+ years of experience in data center, critical facilities, power plant, or large-scale compute operations;

or 6+ years of equivalent experience in those environments.

## Preferred Skills and Experience:

- Hands-on experience with at least one of: Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry and controls (SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus/BACnet).

- Experience in datacenter site reliability, systems engineering, plant operations, or large-scale production operations.

- Experience running acceptance test campaigns for integrated products.

- Experience in AI/ML infrastructure or supercomputing environments: GB200/NVL72 or H100/H200-class GPU fleets, NCCL, InfiniBand/RoCE, Slurm or Kubernetes.

- Fleet firmware and BMC management (Redfish, IPMI), including firmware release validation and staged rollouts.

- Familiarity with data center hardware and plant signals (servers, GPUs, networking, power, cooling) in addition to software telemetry.

- Proven large-scale incident command experience and calm technical leadership on a bridge.

- Experience writing and operating playbooks, runbooks, or standard operating procedures.

- Proficiency in scripting (Python, Bash) for automation and analysis, plus general experience in at least one systems language (C, C++, Java, Go, Rust, or similar).

- Ability to work collaboratively with cross-functional, cross-country, multi-disciplinary teams.

## Skills

### Required
- Mechanical Engineering
- Electrical Engineering
- Computer Engineering
- data center operations
- large-scale compute operations

### Nice to have
- Prometheus/Grafana
- NVIDIA DCGM
- SCADA
- BMS/BAS
- EPMS
- DCIM
- PLCs
- Modbus/BACnet
- datacenter site reliability
- systems engineering
- plant operations
- large-scale production operations
- AI/ML infrastructure
- supercomputing environments
- Kubernetes
- Slurm
- InfiniBand/RoCE
- NCCL
- GB200/NVL72
- H100/H200-class GPU fleets
- firmware and BMC management
- Redfish
- IPMI

---

Source: [Apply at job-boards.greenhouse.io](https://job-boards.greenhouse.io/xai/jobs/5273689007?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)