SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
xAI (SpaceXSI) is seeking a Site Reliability Engineer focused on campus reliability for one of the largest AI training campuses in the world. This role bridges compute, network, storage, power, and cooling infrastructure, requiring calm incident leadership, fleet-scale observability judgment, and the ability to drive reliability work across software and facility boundaries.
Key Responsibilities:
- Own monitoring architecture and signal quality across GPU/host telemetry (DCGM, Prometheus/Grafana) and facility telemetry (BMS, EPMS, DCIM, SCADA, CDU/direct liquid cooling). Treat alert noise as a design failure, not an operator failure.
- Provide SEV command support: technical incident leadership, bridge coordination with NOC, paging/escalation (PagerDuty), and timeline/severity hygiene.
- Run blameless postmortems and drive corrective actions to closure.
- Gate high-risk changes: fleet-scale firmware/BIOS/BMC rollouts (Redfish, canaries, rollback) and SOP/MOP/EOP-governed work on power and cooling systems through disciplined change management.
- Lead cross-functional reliability projects spanning compute, network (InfiniBand/RoCE), storage, and facility signal boundaries, including grey failures that cross those boundaries (NVLink flapping, thermal throttling, power sequencing).
- Partner with Hardware Failure Analysis on systemic failure classes: turn fleet data (MTBF/AFR, repeat-RMA trends) into monitoring, stop-rules, and vendor RCA inputs.
- Build and maintain playbooks, run game days, and keep cross-discipline dependency maps current. Own runbook quality jointly with NOC.
- Define error budgets and availability objectives at campus and service boundaries.
- Participate in on-call rotations and incident response for SEV-class events.
Success is measured by MTTD/MTTR trends, % of SEVs with blameless postmortems and closed actions, alert actionable ratio, recurrence rate of incident classes, monitoring coverage of critical dependencies, and GPU fleet availability lost to infrastructure events.
Requirements:
- Bachelor's degree in Systems Engineering, Computer Science, Electrical Engineering, or related field (or equivalent experience).
- 5+ years in site reliability, systems engineering, plant operations, or large-scale production operations, preferably in power plants, nuclear power plants, HPC, or data center environments.
- Proven large-scale incident command experience and calm technical leadership on a bridge (SRE/NOC bridge or plant control room).
- Demonstrated monitoring and observability design at fleet or campus scale, including alert hygiene, suppression, and signal quality.
- Hands-on experience with at least one of: Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry/controls (SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus/BACnet).
- Experience working across at least two of: compute, network, storage, power, and cooling/facilities telemetry.
- Experience writing and operating playbooks, runbooks, or SOP/MOP/EOPs with 24/7 operations, control room, or NOC partner.
- Proficiency in scripting (Python, Bash) for automation and analysis, plus general experience in at least one systems language (C, C++, Java, Go, Rust, or similar).
- Excellent problem-solving skills with data-driven approach to reliability engineering.
- Ability to work collaboratively with cross-functional teams (NOC, data center operations, hardware failure analysis, infrastructure engineering).
Preferred:
- Experience in power plants, nuclear power plants, or high-consequence industrial control environments (ICS/OT).
- Current or prior work in data centers or facilities like SpaceXSI spanning power, compute, cooling, and related plant systems, including CDU/direct liquid cooling.
- Experience in AI/ML infrastructure or supercomputing: GB200/NVL72 or H100/H200-class GPU fleets, NCCL, InfiniBand/RoCE, Slurm or Kubernetes.
- Fleet firmware and BMC management (Redfish, IPMI), including firmware release validation and staged rollouts.
- Reliability engineering methods: MTBF/AFR analysis, reliability-centered maintenance, grey-failure diagnosis, vendor RCA/RMA partnership.
- Experience with CMMS (e.g., Maximo) and commissioning/integrated systems testing (IST) and operational turnover.
- Hands-on definition and use of SLOs, SLIs, and error budgets at service or campus boundaries.
- Experience running game days, dependency mapping, and closed-loop corrective action programs.
- Familiarity with data center hardware and plant signals (servers, GPUs, networking, power, cooling) in addition to software telemetry.
- Prior work in fast-paced startup or tech company like SpaceXSI.
