SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Glean is a Work AI platform that helps enterprises work smarter with AI. The platform powers intelligent search, AI assistants, and scalable AI agents across a secure, open infrastructure with 100+ enterprise SaaS connectors and flexible LLM choice. Glean has been recognized by Fast Company (Top 10 Most Innovative Companies 2025), CNBC's Disruptor 50, Bloomberg's AI Startups to Watch, Forbes AI 50, and Gartner's Tech Innovators in Agentic AI.
The AI Data Analyst role sits at the center of response quality, competitive benchmarking, and release readiness for Glean's AI product. You will evaluate AI outputs, label and categorize failure modes, maintain high-quality benchmark data, and ensure datasets and evaluations are realistic, consistent, and tied to measurable product quality improvements. This is a highly cross-functional role working closely with QA, Engineering, Product, and the evals team to convert ambiguous quality issues into structured data, actionable feedback, and repeatable quality signals.
Key responsibilities include:
- Run ongoing human labeling for priority AI workflows, including response quality, task success, MCP and tool use, and end-to-end Cowork-style workflows
- Triage bad-query reports, downvotes, escalations, and routed quality issues; categorize failure modes and feed clean analysis back to partner teams
- Perform qualitative analysis to identify recurring patterns such as hallucinations, retrieval failures, tool-use issues, weak grounding, and poor workflow completion
- Follow and improve labeling guidelines and rubrics to ensure consistent, realistic, and useful judgments for model and product improvement
- Maintain high-quality labeled datasets, golden sets, and regression slices; monitor coverage, drift, leakage, and difficulty distribution
- Participate in calibration exercises with human graders and validate LLM-as-a-judge outputs against human labels
- Partner with engineers, PMs, QA, and eval owners on pre/post-change quality reads, benchmark updates, and release-readiness decisions
- Contribute to recurring quality reporting by surfacing top failure modes, coverage gaps, quality shifts, and actionable recommendations
The role is hybrid, requiring 4 days per week in the Bangalore office.
REQUIREMENTS:
- 3–5 years of experience in data labeling, data analysis, QA, or a related field
- Strong analytical judgment and attention to detail, with the ability to apply nuanced rubrics consistently
- Experience evaluating AI-generated outputs or working with NLP, search, recommendation, or other ML systems
- Familiarity with structured qualitative analysis, basic SQL/data retrieval, and spreadsheet-based workflows
- Clear written and verbal communication skills, with the ability to explain findings to both technical and non-technical audiences
- Ability to work independently in ambiguous, fast-moving environments and collaborate effectively across functions
