SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Doctolib is seeking a Senior Inference Engineer to join the ML Platform (TIGER) team in AI & Machine Learning. You will own the self-hosted LLM serving platform that powers Doctolib's AI-driven healthcare products—including Consultation Assistant, Phone Assistant, and Health Companion—while keeping sensitive patient data within Doctolib-controlled infrastructure.
Your core responsibilities include:
• Build shared LLM serving foundations with MLOps engineers: define the serving stack, GPU capacity planning, and deployment automation for open-weight models (Gemma, Mistral, Llama, Qwen, and fine-tuned variants) on Ray Serve/Anyscale, with evolution toward Kubernetes-based orchestration.
• Own the model registry and promotion path: establish versioning, validation, and promotion processes for production models; benchmark performance and output quality under realistic workloads; maintain the curated model catalog with the Agents Platform team.
• Optimize inference performance and efficiency: operate and tune serving engines (vLLM, Triton Inference Server, TensorRT-LLM) to reduce time-to-first-token, inter-token latency, and cost per token through batching, caching, quantization, and parallelism strategies across a GPU fleet (L4 to H200).
• Own production reliability and disaster recovery: define SLOs, run load tests before launches, implement safe rollouts and rollbacks, manage autoscaling, and participate in on-call and incident response.
• Make serving performance and costs measurable: build Datadog dashboards, alerting, and cost reporting to track latency, GPU utilization, and inference spend; identify opportunities to improve resource efficiency.
• Keep the inference stack secure, compliant, and evolving: collaborate with Security and Privacy on network isolation, data residency, and auditability; continuously assess new models, serving engines (e.g., SGLang), and GPU generations.
The tech stack includes Python, vLLM, TensorRT-LLM, Triton, Ray Serve, Kubernetes (GKE), Docker, Terraform, Helm, Kustomize, MLflow, LiteLLM, Braintrust, Datadog, and GCP/AWS cloud providers.
REQUIREMENTS:
• 5+ years of software or computer engineering experience, including hands-on production experience with at least one LLM serving engine (vLLM, SGLang, TensorRT-LLM, TGI, or Triton)—ideally having chosen, introduced, tuned, or scaled one in production, not just run with defaults.
• Proficiency in Python and Terraform, with solid understanding of transformer inference mechanics: prefill vs. decode, KV cache, batching strategies, and why a model is compute- or memory-bound on a given GPU.
• Strong experience running GPU workloads on Kubernetes or managed Ray/Anyscale platform, with a performance engineering mindset: design reproducible benchmarks, read profiler output, back decisions with numbers.
• Comfortable owning production reliability: observability, incident response, and on-call are natural parts of the role.
• Excellent communication and collaboration skills, high agency, pragmatic build-vs-buy approach, comfortable working in a scope still taking shape.
• Fluent in English.
NICE-TO-HAVE:
• Experience with distributed serving frameworks (KServe, llm-d, NVIDIA Dynamo), disaggregated prefill/decode, KV-cache-aware routing, or serving Mixture-of-Experts, multimodal models, or many LoRA adapters on shared base models.
• GPU kernel experience or deep profiling skills (CUDA, Triton, Nsight), or contributions to open-source inference projects.
• Experience in regulated environments (healthcare, finance, public sector) and frameworks such as GDPR, HDS, ISO 27001, or SOC 2.
