SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Mistral is building full-stack AI solutions—from frontier models to developer tools, applications, and compute—partnering with enterprises across finance, manufacturing, defense, healthcare, and the public sector to co-create customized AI systems.
As an Evaluation Engineer, you will own the design and execution of frontier LLM evaluations that shape the next generation of Mistral's flagship models. Your work bridges customer needs and model capabilities, turning real-world industry use cases into rigorous benchmarks.
Key responsibilities include:
• Conduct discovery workshops with customers, industry experts, and cross-functional Mistral teams to identify high-impact use cases (e.g., CAD software updates, chip design verification, financial fraud detection).
• Design novel benchmarks for industry applications and frontier capabilities, including scoring methodologies and reliability checks.
• Build evaluations where AI agents operate within real software environments (CAD tools, simulators, etc.), then run them across Mistral and competitor models to track performance over time.
• Stress-test existing evaluations to identify flaws before they inform critical decisions.
• Source evaluation data from expert vendors; create realistic proxy or synthetic datasets when customer data cannot leave their environment.
• Synthesize findings into clear written reports and presentations for senior leadership and go-to-market teams.
• Develop reusable tools and guides to help other teams build their own evaluations.
You will learn new domains rapidly—one month studying chip design verification, the next understanding banking compliance workflows. The team is new, fast-paced, and autonomous; you'll spend significant time collaborating across Mistral to identify priorities, then independently decide what to build.
Requirements:
• 2+ years of industry experience (software, AI, consulting) or a PhD.
• Software engineering or research background with experience writing production-quality code.
• Direct experience working with customers, industry experts, or cross-functional teams (consulting, solutions engineering, research collaborations).
• Hands-on experience building applications with LLMs.
• Comfort working with large datasets, including building and cleaning them.
• Strong Python proficiency.
• Ability to work in large shared codebases and participate in code reviews.
• Deep understanding of LLM training, inference, and evaluation.
• Excellent communication skills: articulate, clear writing, confident presenting to technical and non-technical audiences.
• Fluent in English.
• High energy, independent, and comfortable with ambiguity. You move proactively, spot what needs doing, and execute with confidence.
• Low-ego, collaborative, and genuinely curious about learning new fields.
Nice to have:
• Published research on LLM evaluations or contributions to open-source evaluation frameworks.
• Experience building datasets, generating synthetic data, developing labeling guidelines, or managing annotation vendors.
• Background in a non-software industry (engineering, finance, science).
