SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Vinyl Equity is seeking an AI Evaluation Engineer to own the quality bar for AI-powered features and the broader product experience. This is a hands-on, blended role combining evaluation engineering for LLM/AI systems with product quality ownership, focused on ensuring trustworthy delivery of features handling real securities and compliance data.
Key Responsibilities:
- Design, build, and maintain evaluation frameworks and benchmark suites for AI/LLM-powered features (document extraction, compliance checks, automated workflows), measuring accuracy, consistency, hallucination rate, and safety.
- Define golden datasets, rubrics, and scoring methodologies using both human-in-the-loop and automated/LLM-as-judge approaches to measure model and product quality objectively.
- Build automated evaluation pipelines integrated into CI/CD that flag regressions before release and produce clear, trackable quality metrics over time.
- Extend evaluation coverage beyond the model layer into full product/QA testing—functional, regression, and end-to-end testing of AI-powered features and surrounding product.
- Partner with product managers and engineers to translate ambiguous quality requirements into measurable, repeatable evaluation criteria.
- Investigate failures and edge cases, perform root-cause analysis across the model/product boundary, and drive fixes with engineering teams.
- Maintain traceability and reporting on evaluation/QA results for compliance-sensitive workflows given the regulated nature of the data handled.
Requirements:
- 5+ years of experience in QA/test engineering, ML evaluation, or related quality-focused engineering roles.
- Hands-on experience testing or evaluating AI/LLM-powered features—building evaluation sets, scoring rubrics, or benchmark harnesses; or strong adjacent automation/QA experience with demonstrated interest in AI evaluation.
- Solid automation/testing fundamentals: scripting (Python and/or TypeScript/Java), API testing, SQL for data validation, and CI/CD integration.
- Comfort working with LLMs and AI tooling directly—prompt engineering, RAG pipelines, or AI-assisted development tools—either as a builder or rigorous evaluator.
- Strong analytical mindset: comfortable defining metrics for fuzzy, subjective quality questions and defending them with data.
- Clear written communication for documenting failure modes and quality bars for both engineers and non-technical stakeholders.
- Bonus: experience in fintech, compliance, or other regulated domains where correctness and auditability are critical.
Nice to Have:
- Experience with vector databases/semantic search (FAISS, ChromaDB, or similar) and RAG evaluation.
- Familiarity with evaluation tooling/frameworks (Ragas, DeepEval, promptfoo, custom LLM-as-judge pipelines).
- Background in Agile/Scrum environments and defect-management tooling (Jira or similar).
