SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Baseten powers mission-critical inference for leading AI companies including Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. The company recently raised $1.5B in Series F funding and is building the platform engineers rely on to ship AI products.
As an Inference Performance Engineer, you will optimize how the world's most demanding AI workloads run faster and more efficiently. You'll work across the full stack—from inference engines and runtimes through scheduling, serving, and routing—applying advanced techniques like prefill/decode disaggregation, speculative decoding, and KV-cache management. Your work directly impacts customer model performance and serving efficiency.
Key responsibilities include:
- Implementing and productionizing cutting-edge inference techniques such as quantization, speculative decoding, KV-cache reuse, chunked prefill, LoRA, guided generation, and custom scheduling/routing algorithms
- Profiling and optimizing inference end-to-end, from kernel launch overhead and memory layout through request scheduling and cache-aware routing
- Converting performance improvements into cost savings by raising tokens-per-GPU-hour, improving utilization, and providing customers with clear latency/throughput/cost tradeoffs
- Bringing up and tuning new model architectures on new hardware rapidly (often within the same week of release)
- Building benchmarking frameworks that measure real-world performance across model architectures, batch sizes, sequence lengths, and hardware configurations
- Contributing upstream to open-source inference engines (vLLM, SGLang, TensorRT-LLM) and partnering with model, infrastructure, and customer-facing teams
Requirements:
- Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, Mathematics, or related field
- Experience with general-purpose programming languages (Python, C++)
- Familiarity with LLM optimization techniques (quantization, speculative decoding, continuous batching)
- Strong familiarity with ML libraries, especially PyTorch, TensorRT, or TensorRT-LLM
- Demonstrated interest and experience with LLMs
- Deep understanding of GPU architecture
Nice to have:
- Proficiency in software performance optimization for large language models
- Contributions to vLLM, SGLang, TensorRT-LLM, or similar inference engines
- Experience with large-scale distributed serving (autoscaling, load balancing, multi-region/multi-cloud)
- GPU kernel optimization experience (CUDA, Triton, CUTLASS)
- Production experience with quantization or speculative decoding
- Strong software engineering principles and proven track record deploying AI/ML inference solutions
