SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Mistral is building full-stack AI solutions spanning frontier models, developer tools, applications, and compute infrastructure. The company partners with enterprises across finance, manufacturing, defense, healthcare, and the public sector to co-create customized AI systems.
As a Research Engineer on the Evaluation Platform team, you will own the infrastructure that enables Mistral's science teams to measure model quality reliably and at scale. Your work directly impacts which models, checkpoints, and recipes ship to production.
Key responsibilities:
- Design and build systems that maintain reproducibility and comparability of evaluation results as models, benchmarks, and code evolve over time
- Orchestrate evaluations at scale across GPU clusters, handling everything from model serving to scoring
- Create APIs and dashboards that make evaluation results accessible, explorable, and trustworthy for researchers
- Implement quality checks to catch broken or noisy evaluations before they mislead research decisions
- Support evaluation of complex model behaviors including agentic, multi-turn, and tool-using capabilities
- Partner closely with research teams to translate new evaluation requirements into robust, shared tooling
You will work in a distributed, collaborative environment with teams across Europe, North America, Asia, and the Middle East. The team values creativity, low ego, and team spirit.
Requirements:
- Master's degree or PhD in Computer Science, or equivalent professional experience
- 4+ years building production-grade software, ideally with experience in large-scale ML codebases or distributed systems
- Excellent Python skills and strong software-design fundamentals (testing, code review, CI/CD)
- Hands-on experience running workloads on GPU clusters (Slurm, Kubernetes, Ray, or similar)
- Familiarity with LLM inference and evaluation methodologies
- Product mindset with researchers as your primary users
- Self-starter mentality with collaborative, low-ego approach
Nice to have:
- Experience building or maintaining evaluation harnesses or benchmarks
- Hands-on work with inference engines (vLLM, SGLang)
- Experience with agentic or reinforcement learning environments
- Statistics expertise for experimentation (variance estimation, significance testing)
- Open-source contributions to ML tooling
