SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
xAI (SpaceXSI) is seeking a Supercompute Operations Engineering professional to own 100% GPU uptime across their Colossus supercomputer facilities. This role is central to the company's mission of building Super Intelligence infrastructure that powers Grok models and applications.
You will own end-to-end operations and incident response for GPU and facility systems 24/7, working across both compute and facility domains. Key responsibilities include:
- Monitoring architecture and automation: Own monitoring across GPU/host telemetry (DCGM, Prometheus/Grafana) and facility telemetry (BMS, EPMS, DCIM, SCADA, CDU/liquid cooling).
- Hardware change management: Incorporate physical, firmware, and software changes to compute hardware.
- 24/7 incident response: Lead mitigation and control authority from customer impact back to facility level.
- Site sustainment: Resolve issues impacting rack-level compute and facility infrastructure; disposition issues across Site Operations and Facility Operations.
- Deployment leadership: Lead acceptance and deployment testing campaigns for new data hall bring-up.
- Automation development: Build production automation for new datahall deployment, alerting, and automated incident response.
- Operational documentation: Develop and maintain SOPs, planning, and procedures for facility and site technicians.
- Cross-functional reliability: Lead projects spanning compute, network (InfiniBand/RoCE), storage, and facility systems, including grey failures.
- Root cause analysis: Analyze fleet data (MTBF/AFR, RMA trends) to drive proactive alerting, vendor RCAs, and supplier corrective actions.
You will work directly with technicians in data halls and the field, enabling them to operate more efficiently and safely while maximizing GPU uptime.
REQUIREMENTS:
- Bachelor's degree in Mechanical, Electrical, Computer Engineering, or related scientific discipline, plus 2+ years in data center, critical facilities, power plant, or large-scale compute operations; OR 6+ years equivalent experience.
PREFERRED QUALIFICATIONS:
- Hands-on experience with Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry/controls (SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus/BACnet).
- Data center site reliability, systems engineering, plant operations, or large-scale production operations experience.
- Acceptance test campaign experience for integrated products.
- AI/ML infrastructure or supercomputing experience: GB200/NVL72 or H100/H200-class GPU fleets, NCCL, InfiniBand/RoCE, Slurm, or Kubernetes.
- Fleet firmware and BMC management (Redfish, IPMI), including firmware release validation and staged rollouts.
- Data center hardware and plant signals knowledge (servers, GPUs, networking, power, cooling).
- Large-scale incident command experience and calm technical leadership (SRE/NOC bridge or plant control room).
- Playbook/runbook/SOP writing and operations experience.
- Scripting proficiency (Python, Bash) and systems language experience (C, C++, Java, Go, Rust, or similar).
- Cross-functional, multi-disciplinary team collaboration.
