Description
SpaceXSI is building Super Intelligence to help humanity understand the universe.
The team owns 100% GPU uptime, end to end, 24/7, across both compute and facility. It owns incident response and the engineering behind it, holds central control authority from the customer back to the facility, runs day-to-day sustainment of live datahalls, and leads deployment engineering for new datahall bring-up. The team will also build the automation for continuous response improvement.
Responsibilities:
- Own monitoring architecture, alerting, signal quality, and automation of monitoring across GPU and host telemetry (DCGM, Prometheus/Grafana) and facility telemetry (BMS, EPMS, DCIM, SCADA, CDU / direct liquid cooling).
- Own compute hardware change incorporation - physical, firmware, software.
- Be part of 24/7 incident response, mitigation, and control authority - end to end - from customer impact back to the facility.
- Own day-to-day site sustainment operations, including resolving issues impacting rack-level compute and facility infrastructure.
- Disposition daily issues across both Site Operations (rack compute) and Facility Operations (power and cooling).
- Lead acceptance and deployment testing campaigns to bring new data halls fully online.
- Develop production automation for new datahall bring-up, automation behind alerting and automated incident response.
- Develop and maintain standard operating procedures and planning for facility and site technicians.
- Own and document the live, changing state of datahall operation as new halls come online.
- Enable Site Operations and Facility Operations technicians to do their day to day jobs more efficiently and safely to increase GPU uptime.
- Create predictive and responsive automation to alerts and incidents to enable your response to be faster and more impactful.
- Lead cross-functional reliability projects spanning compute, network (InfiniBand/RoCE), storage, and facility signal boundaries.
- Truly root cause systemic and design failure classes across hardware and integrated facility and operations systems.
Basic Qualifications:
- Bachelor's degree in Mechanical, Electrical, or Computer Engineering or a related scientific discipline and 2+ years of experience in data center, critical facilities, power plant, or large-scale compute operations;
or 6+ years of equivalent experience in those environments.
Preferred Skills and Experience:
- Hands-on experience with at least one of: Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry and controls (SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus/BACnet).
- Experience in datacenter site reliability, systems engineering, plant operations, or large-scale production operations.
- Experience running acceptance test campaigns for integrated products.
- Experience in AI/ML infrastructure or supercomputing environments: GB200/NVL72 or H100/H200-class GPU fleets, NCCL, InfiniBand/RoCE, Slurm or Kubernetes.
- Fleet firmware and BMC management (Redfish, IPMI), including firmware release validation and staged rollouts.
- Familiarity with data center hardware and plant signals (servers, GPUs, networking, power, cooling) in addition to software telemetry.
- Proven large-scale incident command experience and calm technical leadership on a bridge.
- Experience writing and operating playbooks, runbooks, or standard operating procedures.
- Proficiency in scripting (Python, Bash) for automation and analysis, plus general experience in at least one systems language (C, C++, Java, Go, Rust, or similar).
- Ability to work collaboratively with cross-functional, cross-country, multi-disciplinary teams.