SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
FurtherAI is building AI agents for the insurance industry, a trillion-dollar market still driven by manual work. The company has raised $30M+ from top-tier investors (Andreessen Horowitz, YC, Nexus, South Park Commons), achieved 100%+ YoY growth, and partners with top 10 insurance carriers, brokers, and Fortune 100 companies. Both founders are engineers, and the company is deeply technical with a founder's mindset culture.
In this role, you will own evaluation and continuous improvement of AI agent quality across FurtherAI's platform. You'll build evaluation frameworks and infrastructure to measure accuracy, reliability, and quality of AI agents across workflows, document processing, and computer use. Key responsibilities include:
- Designing and building evaluation frameworks for multi-step agent trajectories, tool use, reasoning, and end-to-end task completion
- Developing golden datasets, benchmarks, and regression suites in collaboration with Forward Deployed Engineers and domain experts
- Creating LLM-as-judge evaluations, deterministic graders, human evaluation workflows, and quality rubrics
- Integrating offline and online evaluations into development and release processes to enable engineers to measure improvements and catch regressions
- Benchmarking frontier and open-weight models across accuracy, latency, cost, and reliability to inform model selection and production releases
- Building continuous improvement loops by converting production failures, customer feedback, and agent traces into new evaluation cases
Representative projects include building an end-to-end evaluation framework for multi-step insurance workflows, creating a reusable library of golden datasets and benchmarks, developing automated benchmarking systems for frontier and open-weight models, and building production evaluation pipelines that detect quality regressions and identify failure patterns.
The team includes founders and engineers from Apple AI Research, Brex, Coinbase, and Uber; 4 YC founders and 6 ex-founders; and graduates of Stanford, Berkeley, IITs, and Cornell. The company operates 5 days per week in-office in San Francisco with lunch and dinner provided.
REQUIREMENTS:
- 5+ years of software engineering experience with strong hands-on production AI/ML systems building
- Experience building evaluation frameworks, LLM-based applications, model benchmarking systems, or applied AI infrastructure
- Strong understanding of modern LLMs, agent architectures, evaluation methodologies, and model performance tradeoffs
- Strong Python engineering skills and ability to build reliable, production-grade systems
- Ability to independently own ambiguous technical problems from research through production
- Clear communication skills, both verbal and written
- Bonus: LLM-as-judge, human evaluation systems, agent trajectory evaluations, or large-scale benchmarking experience
- Bonus: Open-weight models, fine-tuning, post-training, or model routing experience
- Bonus: B2B SaaS background and 0-to-1 experience
- Above all: drive, grit, and ownership mindset
