SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Salary: USD 250,000 - 320,000 / annual
Together AI is seeking a Staff AI Infrastructure Engineer to own the production reliability and operational excellence of their Kubernetes-native GPU inference fleet control plane. This is a high-impact role focused on keeping the platform's manifest-driven API and controllers working reliably at scale, ensuring that inference teams can declare their needs (clusters, model deployments, capacity changes) and have them reconciled flawlessly in production.
You will be responsible for triaging and resolving complex distributed systems issues—InfiniBand/RDMA stalls during KV cache exchanges between prefill and decode nodes, hero-model and LoRA-serving faults that don't surface in generic health checks, and extending the automated self-healing system so each new failure mode becomes a permanent fix rather than a recurring incident. Your core mission is maintaining the decoupling between the control plane and the inference layer even when things break, while keeping the fleet efficient without sacrificing stability.
Key responsibilities include:
- Operating the provisioning state machine in production: owning the host lifecycle pipeline (discovery, bring-up, GPU driver/CUDA validation, health checks, decommission/RMA), triaging stuck or failed state transitions at fleet scale, and feeding real failure patterns back into state machine design.
- Triaging and extending automated self-healing: diagnosing new failure modes live, driving fixes to production, and turning them into permanent detection/remediation plugins.
- Owning end-to-end reliability: holding SLOs and error budgets for provisioning and fleet health, leading incident response, and driving systemic fixes rather than hotfixes.
- Partnering with the inference/ML platform team: bringing production-side knowledge of how real cluster topologies, interconnects, and scheduling constraints fail in the field back into platform abstractions.
- Collaborating with ML researchers, engineers, and product managers to bring new technologies into production.
- Building tools for visibility into bottlenecks and instability sources, then designing and implementing solutions.
- Leading reliability cross-functionally: building coalitions across infra, ML platform, and product teams around shared reliability priorities, managing stakeholders through incidents, and representing the reliability perspective in planning.
You will own SLOs and error budgets, lead incident response, and turn postmortem learnings into actual changes in controllers and automation. A product mindset applies: the on-call experience, alert clarity, and automated remediation logic are all things you ship and iterate on.
Requirements:
- Experience operating a provisioning or host-lifecycle pipeline in production, comfortable troubleshooting stuck or failed state transitions at fleet scale, and translating real failure patterns into state machine improvements.
- Deep experience owning production reliability for large-scale distributed systems; GPU/ML-serving infrastructure experience is a strong plus.
- Hands-on experience troubleshooting InfiniBand/RDMA networking issues, ideally in multi-node training or disaggregated-inference contexts—comfortable reading IB port counters and subnet manager logs, not just application metrics.
- Familiarity with operational realities of multi-tenant model serving (LoRA/adapter-based serving) and running differentiated SLOs across model pools of different priority.
- Demonstrated track record of eliminating operational toil through automation and tooling you built yourself, not just process or documentation.
- Hands-on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node-level health/remediation systems.
- Comfort with event-driven architectures, backpressure/circuit-breaking design, and treating observability (metrics, logs, traces) as built-in, not added after incidents.
- Software-engineering mindset first: you write and ship production code, and think about GPU infrastructure as software to be engineered, not a fleet to be manually operated.
About Together AI
AI / Data / Infrastructure — cloud platform for open-source and generative AI model training and inference.
