SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Delta Exchange is a cryptocurrency derivatives exchange with ~$10 billion in daily traded volume, serving global traders since 2018. The company operates three AI products used by live traders: a customer support chatbot, an API Copilot that generates and executes trading scripts, and an MCP server.
This Senior QA Engineer role owns the quality of these AI products and is responsible for measuring it objectively. Unlike traditional QA, testing probabilistic AI systems requires fundamentally different approaches—the same input produces different outputs on every run, incorrect answers can be fluent and convincing, and prompt changes in one flow can degrade another without visible signals.
The core responsibility is establishing what "correct" means for these systems, building datasets and scoring mechanisms that measure quality, and operating the release gate that prevents regressions from reaching customers. This is an emerging discipline with no established playbook; you will define the methodology as you build it with high autonomy and ownership.
Key responsibilities include:
- Building and maintaining golden datasets for AI products, traced from production or generated and verified against documented sources of truth
- Designing layered scoring for AI outputs: deterministic rules where behavior can be asserted, and model-graded evaluation where it cannot
- Managing evaluation runs end-to-end: scheduling and executing against release candidates, comparing results against baselines, and triaging failures
- Operating the release gate that determines whether prompt or model changes are approved for production
- Identifying failure modes ahead of customers (hallucination, incorrect tool selection, context loss, PII exposure)
- Investigating production traces to distinguish retrieval failures from generation failures from tool failures
- Converting findings into actionable engineering evidence and permanent regression coverage
- Defining and reporting quality metrics for AI surfaces and driving improvement
The founding team includes IIT and ISB graduates with prior experience at Citibank, UBS, GIC, and serial entrepreneurs from TinyOwl and Housing.com. The company is funded by top crypto funds (Sino Global Capital, CoinFund, Gumi Cryptos) and crypto projects (Aave, Kyber Network).
REQUIREMENTS:
- Minimum 1 year owning quality for AI products in a lead or primary-owner capacity (chatbots, conversational assistants, code/content generation, agentic systems)
- Direct hands-on experience with prompts, tool and function schemas, and model behavior
- Demonstrated hands-on experience building or operating an evaluation harness for an AI system (in-house or using frameworks like DeepEval, RAGAS, Promptfoo)
- 4-6 years of QA/SDET experience
- Working proficiency in Python (evaluation platform, trace analysis, and internal tooling are Python-based)
- Ability to analyze traces and tool calls using observability tooling (Opik, Langfuse, LangSmith, or equivalent) to determine root cause
- Strong API testing experience (majority of surface under test is API-level)
- Sufficient engineering ability to build own tooling and automation
- Clear written communication for documenting findings in actionable form
BONUS:
- Experience with trading or exchange platforms (futures, options, crypto derivatives, margin, settlement flows)
- Domain knowledge in crypto/trading (can be picked up on the job)
