SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Delta Exchange is a cryptocurrency derivatives exchange with $10 billion in daily traded volume, serving global traders since 2018. The company operates three AI products: a customer support chatbot, an API Copilot that generates and executes trading scripts, and an MCP server. This Senior QA Engineer role owns the quality of these AI products and is responsible for measuring it objectively.
Unlike traditional QA roles that use AI to accelerate testing, this position focuses on testing AI systems themselves—a fundamentally different challenge. The core problem: the same input produces different outputs on every run, incorrect answers can be fluent and convincing, and prompt changes in one flow can degrade another without visible signals. You will establish what "correct" means for probabilistic systems, build datasets and scoring mechanisms to measure quality, and operate the release gate that prevents regressions from reaching customers.
Key responsibilities include:
- Building and maintaining golden datasets for AI products, traced from production or generated and verified against documented sources of truth
- Designing layered scoring for AI outputs: deterministic rules where behavior can be asserted, and model-graded evaluation where it cannot
- Managing evaluation runs end-to-end: scheduling and executing against release candidates, comparing results against baselines, and triaging failures
- Operating the release gate that determines whether prompt or model changes are approved for production
- Identifying failure modes ahead of customers (hallucination, incorrect tool selection, context loss, PII exposure)
- Investigating production traces to distinguish retrieval failures from generation failures from tool failures
- Converting findings into actionable engineering evidence and permanent regression coverage
- Defining and reporting quality metrics for AI surfaces and driving improvement
The founding team includes IIT and ISB graduates with prior experience at Citibank, UBS, and GIC. The tech co-founder previously co-founded TinyOwl and Housing.com. Delta is funded by top crypto funds (Sino Global Capital, CoinFund, Gumi Cryptos) and crypto projects (Aave and Kyber Network).
Requirements:
- At least 1 year owning quality for AI products in a lead or primary-owner capacity (chatbots, conversational assistants, code/content generation, agentic systems). Must have worked directly with prompts, tool/function schemas, and model behavior, not around them.
- Demonstrated hands-on experience building or operating an evaluation harness for an AI system (in-house or using frameworks like DeepEval, RAGAS, Promptfoo). Must be able to describe what the harness measured, results produced, and decisions informed.
- 4-6 years of QA/SDET experience
- Working proficiency in Python (evaluation platform, trace analysis, and internal tooling are Python-based)
- Ability to analyze traces and tool calls using observability tooling (Opik, Langfuse, LangSmith, or equivalent) to determine root cause
- Strong API testing experience (majority of surface under test is API-level)
- Sufficient engineering ability to build own tooling and automation
- Clear written communication for documentation that engineering can act on directly
Bonus: Experience with trading or exchange platforms, including familiarity with futures, options, crypto derivatives, margin, or settlement flows.
