SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Salary: USD 200,000 - 280,000 / annual
Glean is a Work AI platform that helps enterprises work smarter with AI. The company has evolved from enterprise search into a full-scale AI ecosystem, powering intelligent search, AI assistants, and scalable AI agents on a secure, open platform. With over 100 enterprise SaaS connectors, flexible LLM choice, and robust APIs, Glean enables organizations to govern, scale, and customize AI across their business without vendor lock-in.
The Data and Evals team owns evaluation science for Glean's AI products. The core challenge in building reliable, useful AI is no longer writing code—it's defining what good looks like and measuring how well products deliver it. This requires high-quality data and evals that teams can trust.
In this role, you will build evalsets, agentic LLM-judges, and benchmarks that capture what good looks like, along with the platform and infrastructure that run evals reliably at scale. You'll create enterprise simulation environments where assistants and agents are tested on realistic multi-step tasks. Starting from real production usage, you'll turn learnings into evals that guide improvements across the entire stack: agent harnesses, tool use, skill iteration, model routing, and post-training of task-specific models.
Key responsibilities include:
- Building evalsets and benchmarks from expert-curated examples, production data, synthetic data, and past failures, covering edge cases and key user slices
- Building and scaling the eval platform and agent observability: experiment pipelines, trace capture and replay, and dashboards with reproducible, affordable runs
- Building agentic LLM verifiers for correctness, groundedness, citation quality, and task success, calibrated against human labels using clear rubrics
- Running evals at every stage (CI, pre-release, sampled production traffic) and reporting results with confidence intervals
- Building enterprise simulation environments with simulated users and realistic company data to test multi-step agent tasks
- Diagnosing failures by slice and trace, turning eval results into concrete fixes while weighing quality against latency and cost
You'll partner with R&D, product, and GTM teams to turn findings from evals into better products and customer outcomes.
REQUIREMENTS:
- 5+ years of software engineering experience with strong Go and/or Python and backend fundamentals: APIs, data modeling, testing, and distributed pipelines. TypeScript is a plus.
- Experience building datasets and benchmarks for LLM use-cases, RAG, or agents, including evalset curation and synthetic data generation
- Familiarity with eval methodologies and their tradeoffs: deterministic metrics, LLM-as-a-judge, pointwise vs pairwise grading, human labeling, and judge calibration
- Experience building eval or experiment infrastructure that runs reliably at scale, including CI/CD integrations and observability tooling like OpenTelemetry
- Strong analytical skills: ability to dig into traces and failure cases, write SQL, and use data and metrics to explain quality movements
- Nice to have: experience with simulation environments and/or model post-training techniques
