SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Alpaca is a US-headquartered global leader in agent-first brokerage infrastructure serving hundreds of financial institutions across 40 countries. The company provides institutional-grade APIs for stocks, ETFs, options, crypto, fixed income, and 24/5 trading, supporting over 10 million brokerage accounts. Backed by $400 million in funding from top-tier investors including Portage Ventures, Spark Capital, and Y Combinator, Alpaca operates a globally distributed team of 400+ members.
You will join as a Senior Data Scientist focused on AI Evaluation, owning the design and implementation of how Alpaca measures whether its models and agents are correct. This is a senior individual contributor role with significant ownership over a greenfield evaluation practice. You will turn ambiguous quality questions into ground truth, scoring methods, and evaluation loops that the company can trust, then use those results to drive system improvements.
Key responsibilities include:
- Design AI evaluations by defining ground truth, metrics, and scoring methods for models and agents
- Build repeatable evaluation loops that track quality over time and catch regressions before release
- Partner with engineering and analytics engineering teams to operationalize evaluation harnesses
- Translate evaluation results into actionable recommendations for system improvements
- Establish quality standards, evaluation guidelines, documentation, and review practices
- Mentor teams and foster a culture of measurable AI quality across the organization
You will work cross-functionally with Product, Engineering, Analytics Engineering, and business stakeholders to define what "good" looks like and close the loop so evaluations drive iteration.
Requirements:
- 6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation
- Track record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or experimentation)
- Strong statistical and ML foundation; ability to treat evaluations as experiments (sample sizing, confidence intervals, significance testing, handling non-determinism)
- Proficiency in Python and SQL, with experience evaluating models in production environments
- Strong judgment in defining quality metrics and ground truth for ambiguous outputs
- Excellent communication and cross-functional collaboration skills
- Strong problem-solving ability in fast-paced, greenfield environments
- Quantitative degree is a plus; equivalent industry experience equally welcome
Nice to have:
- Hands-on LLM/agent evaluation in production, including eval harnesses, LLM-as-judge calibration, and CI regression gates
- Experience evaluating text-to-SQL, analytics agents, or systems where correctness is verifiable against data
- Background in fintech, brokerage, or domains where incorrect answers have real business or risk consequences
- Fluency with AI tools in research and engineering workflows
