SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Mistral is building full-stack AI solutions spanning frontier models, developer tools, applications, and compute infrastructure. The company partners with enterprises across finance, manufacturing, defense, healthcare, and the public sector to co-create customized AI systems.
The Inference Foundation team owns the core of Mistral's inference stack—the inference engine and its orchestration—serving production models while maintaining strict quality-of-service targets. This hybrid role spans production LLM serving, engine and platform development, and capacity engineering.
You will work on three interconnected problems:
1. **Optimize the inference stack at scale**: Develop and fix the core inference engine and orchestrator, including feature selection, configuration, and tuning for maximum performance. Own the release process for the serving stack, ensuring validated, regression-free releases through automated performance gates and progressive rollout. Drive improvements and fixes upstream to open-source engines when appropriate.
2. **Make capacity elastic**: Optimize serving efficiency across the fleet by driving down pod startup time, tackling cold-cache regressions on scale-up, and implementing smarter caching and offloading. Optimize and maintain the optimal serving topology—overlapping communication and transfers with computation, ensuring optimal placement, connectivity, and routing.
3. **Power frontier model training**: Build the serving infrastructure that powers RL and post-training loops for frontier models. Optimize inference performance across the full spectrum of workloads to keep training loops running at full speed.
You will be comfortable debugging across the full stack—CUDA/NCCL, kernels, containers, networking, and storage—and will have hands-on experience with production ML/LLM services at scale.
**Requirements:**
- Experience building and running ML/LLM services at scale with clear latency and availability targets
- Hands-on experience with inference engines (vLLM, SGLang, TensorRT-LLM, or similar)
- Solid grasp of inference internals: prefill vs. decode, KV-cache behavior, batching, scheduling, speculative decoding, parallelism strategies
- Familiarity with distributed and disaggregated serving architectures
- Comfortable debugging across the full stack—CUDA/NCCL, kernels, containers, networking, storage
- Python for systems tooling and backend services; PyTorch
- Kubernetes for running infrastructure at scale
- GPU and networking fundamentals: CUDA runtime, NCCL, InfiniBand/RDMA
**Nice to have:**
- Demonstrated vLLM/sglang expertise—upstream contributions or track record in demanding production environments
- Hardware-aware optimization for various model architectures
- Experience serving MoE models at scale (expert parallelism, expert placement/load balancing)
- CUDA/Triton kernel development; Nsight Systems/Compute profiling
- Rust and/or C++ in production systems
