SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Fundamental is an AI research lab developing NEXUS, a Large Tabular Model (LTM) for enterprise decision-making. The company has raised $275M in funding and is trusted by Fortune 100 companies.
You will lead the Evaluations team, owning how NEXUS is measured and building the shared evaluation platform that engineering, research, and Applied AI teams rely on. This is a player-coach role where you will hire and manage a small team while remaining hands-on with code, experiment design, and methodology.
Key responsibilities include:
- Building and maintaining a shared evaluation platform with models, curated datasets, and benchmarking capabilities for independent team use
- Defining evaluation standards for metrics, data splits, leakage prevention, calibration, uncertainty, and benchmark contamination
- Creating reproducible pipelines with versioned inputs and artifacts, integrating regression checks into research and release workflows
- Benchmarking NEXUS against competing approaches using fair tuning budgets, data access, compute, and latency measurement protocols
- Supporting Applied AI's customer POC evaluations with tooling, methodological guidance, and analysis
- Measuring predictive quality, latency, and cost across deployment configurations, task types, and dataset characteristics
- Translating findings into research priorities, release recommendations, and evidence-backed customer improvement plans
- Hiring, developing, and managing the team while staying technically involved
Requirements:
- Experience owning evaluation for tabular ML systems used in production or consequential customer decisions
- Strong statistical judgment: selecting metrics and validation schemes, estimating uncertainty, comparing models across datasets, accounting for repeated experimentation
- Practical experience identifying leakage in preprocessing, feature construction, joins, temporal dependencies, and related entities across splits
- Strong Python and SQL skills, familiarity with scikit-learn and gradient-boosted trees, experience building reliable ML tooling or platforms used by other teams
- Experience designing fair model comparisons, including hyperparameter search, resource budgets, and end-to-end latency measurement
- Prior people management experience including hiring, technical coaching, and performance feedback, while remaining technically involved
- Clear written and spoken communication with researchers, engineers, and customer data scientists, including willingness to challenge unsupported claims
Nice to have:
- Experience evaluating tabular foundation models or AutoML systems
- Experience measuring how model optimizations affect predictive quality and inference performance
- Experience with relational or multi-table data, Snowflake or Databricks environments
- Experience evaluating automated or agent-driven ML workflows, including failures hidden by aggregate metrics
