SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Rubrik is seeking a Senior AI Evals Engineer to design, develop, and execute robust evaluation frameworks and test strategies for AI agents built using the Google Agent Development Kit (ADK) and similar agentic frameworks. This is a hands-on technical role that bridges AI development and reliable deployment, with a strong focus on Responsible AI and Safety Evals, including adversarial testing, red-teaming, bias detection, and toxicity measurement.
The role is approximately 70% automation and 30% manual testing, sitting within the Applied AI: Test Engineering team as a key technical authority.
Key responsibilities include:
- Design and build end-to-end agent evaluation frameworks for assessing correctness, latency, cost, safety, and task completion rates
- Build synthetic testing environments and simulation strategies to stress-test AI agents under diverse real-world conditions, with focus on Google ADK and Vertex AI/Gemini models
- Curate domain-specific evaluation datasets for Rubrik's product verticals (data security, SaaS protection, cyber recovery, compliance)
- Establish eval-as-a-service platform enabling self-serve testing against standardized and custom benchmarks
- Implement CI/CD-integrated evaluation pipelines for regression detection and drift monitoring before production deployment
- Define evaluation taxonomies and rubrics for agentic workflows (tool use accuracy, multi-step reasoning, hallucination rate, policy adherence, error recovery)
- Build production observability and real-time monitoring for agent quality metrics
- Run structured model comparisons to inform model selection decisions across use cases
- Provide clear, actionable quality and safety reports to stakeholders with focus on risk metrics and mitigation recommendations
- Define comprehensive end-to-end QA strategies (functional, integration, regression, UAT) for conversational and goal-oriented AI agents
- Develop and maintain detailed test artifacts covering agent features, tool invocation, memory persistence, and reasoning accuracy
- Identify, document, prioritize, and track defects and performance regressions using Jira
- Evaluate new technologies and run proofs-of-concept to influence platform roadmap
- Partner with policy and legal teams to ensure evaluations cover regulatory compliance and AI governance requirements
- Provide technical leadership including solutioning, design reviews, code reviews, pair programming, and mentoring
- Collaborate cross-functionally with ML engineers, product teams, platform engineers, and security teams
Requirements:
- Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience
- 8+ years of overall software test engineering experience, with 5+ years focused on test engineering, QA automation, or developer tooling, and demonstrated ability to build evaluation systems across multiple AI/ML domains, conversational agents, or LLMs
- 2+ years working with ML/AI systems, building, testing, or evaluating models in production environments, including hands-on experience with LLM-based applications, prompt engineering, RAG pipelines, or agentic workflows
- 2+ years as a senior/lead, owning evaluation strategy, quality frameworks, and technical decisions for complex AI agent systems in an enterprise product environment
- 1+ years designing or operating evaluation frameworks specifically for generative AI/LLM agents, including benchmarking, red-teaming, and production quality monitoring
- Deep hands-on experience with LLM evaluation frameworks and libraries (LangSmith, DeepEval, LLM as a Judge, Promptfoo, RAGAS, Hugging Face evaluation tooling)
- Strong understanding of agentic AI patterns (tool use, function calling, multi-step planning, ReAct/CoT reasoning chains, RAG)
- Production experience building automated, CI-integrated evaluation infrastructure
- Proficiency in Python with strong scripting and tooling skills; experience with pytest, async patterns, and data processing at scale
- Familiarity with Google Cloud Platform (GCP) or cloud services relevant to AI/ML (e.g., Vertex AI) and experience integrating testing into MLOps workflows
- Statistical literacy to design experiments, interpret eval metrics with appropriate confidence, and avoid common pitfalls
- Enterprise product sensibility understanding how AI quality translates to customer trust in B2B/enterprise contexts
- Excellent communication ability to present eval results and quality trade-offs to engineering leadership and articulate risks to non-technical audiences
- Hands-on experience with CI/CD pipelines and Git-based workflows, including automated testing, static analysis, and deployment automation
- Required tech stack: Python (must), TypeScript, Java (optional); Selenium, Playwright; Anthropic Claude, OpenAI, Azure OpenAI, open-source models; DeepEvals (must), LLM as a Judge (must), Promptfoo/RAGAS/LangSmith; LangGraph, Claude Agent SDK; DevOps basics, Kubernetes, CI/CD (Jenkins/GitHub Actions), observability stacks (Datadog/Grafana)
Preferred qualifications include deep learning/NLP knowledge, data analysis experience with Pandas and SQL, AI ethics and policy background, experience with GIGO testing tools, testing complex APIs for tool-integration and function-calling scenarios, prompt injection and data poisoning skills, and relevant certifications in AI/ML or cloud platforms.
