SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Reflection AI is building open foundational models and seeking a Member of Technical Staff to design and operate the evaluation infrastructure that guides model development and release decisions.
You will own the full stack of evaluation systems: from designing clear APIs and data models for evaluation tasks and datasets, to building distributed execution infrastructure for evaluating model checkpoints throughout training, to creating researcher-facing dashboards and debugging tools. The role spans evaluation authoring, orchestration, performance optimization, and operational monitoring.
Key responsibilities include:
- Design and evolve evaluation abstractions with clear APIs and data models that make it straightforward to add new evaluations and extend existing workflows.
- Build scalable, reliable distributed systems for evaluating model checkpoints with robust orchestration, observability, and failure recovery.
- Optimize inference and evaluation performance through profiling and system tuning to improve throughput, GPU utilization, and time to results.
- Establish evaluation correctness and reproducibility through validation tests, dataset versioning, and provenance tracking.
- Create intuitive interfaces and dashboards for inspecting model outputs, comparing checkpoints, and debugging unexpected behavior.
- Operate and monitor evaluation workloads as training progresses, triage failures, and develop diagnostics to distinguish model quality changes from infrastructure issues.
- Partner directly with researchers to identify bottlenecks, prioritize high-impact improvements, and turn bespoke workflows into self-service tools.
You will work in a talent-dense research lab environment with direct access to researchers and influence over the direction of open AI development.
REQUIREMENTS:
- Strong proficiency in Python, C++, or Rust with deep understanding of concurrency, memory behavior, and performance.
- Proven experience designing, building, and operating reliable, maintainable distributed systems with sound judgment about interfaces, failure modes, and architectural tradeoffs.
- Working knowledge of model inference and evaluation, including how implementation and configuration choices affect measured model quality.
- Ability to investigate complex problems across system boundaries, follow evidence, and identify root causes.
- Strong product judgment and experience translating technical user needs into clear APIs and usable tools.
- Comfort owning ambiguous problems end-to-end, prioritizing competing needs, and balancing rapid delivery with long-term reliability.
- Attention to detail and persistence to investigate subtle discrepancies in model outputs and system behavior.
