SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Delta Exchange is a cryptocurrency derivatives exchange with $10 billion in daily traded volume, serving global traders since 2018. The company operates three AI products used by live traders: a customer support chatbot, an API Copilot that generates and executes trading scripts, and an MCP server.
This Senior QA Engineer role owns the quality of these AI products and is responsible for measuring it objectively. Unlike traditional QA, testing probabilistic AI systems requires fundamentally different approaches. The same input produces different outputs on every run, incorrect answers can be fluent and convincing, and prompt changes in one flow can degrade another without visible signals.
The core responsibility is establishing what "correct" means for these systems, building datasets and scoring mechanisms that measure quality, and operating the release gate that prevents regressions from reaching customers. This is an emerging discipline with no established playbook—you will define the methodology as you build it with high autonomy and ownership.
Key Responsibilities:
- Build and maintain golden datasets for AI products, traced from production or generated, verified against documented sources of truth
- Design layered scoring for AI outputs: deterministic rules where behavior can be asserted, and model-graded evaluation where it cannot
- Manage evaluation runs end-to-end: schedule and execute against release candidates, compare results against baselines, triage failures
- Own and operate the release gate determining whether prompt or model changes are approved for production
- Identify failure modes ahead of customers: hallucination, incorrect tool selection, context loss, PII exposure
- Investigate production traces to distinguish retrieval failures from generation failures from tool failures
- Convert findings into actionable engineering evidence and permanent regression coverage
- Define and report quality metrics for AI surfaces, driving improvement
The founding team includes IIT and ISB graduates with backgrounds at Citibank, UBS, GIC, and serial entrepreneurs from TinyOwl and Housing.com. The company is funded by top crypto funds (Sino Global Capital, CoinFund, Gumi Cryptos) and crypto projects (Aave, Kyber Network).
Requirements:
- At least 1 year owning quality for AI products in a lead or primary-owner capacity (chatbots, conversational assistants, code/content generation, agentic systems)
- Direct hands-on experience with prompts, tool and function schemas, and model behavior
- Demonstrated hands-on experience building or operating an evaluation harness for an AI system (DeepEval, RAGAS, Promptfoo, or equivalent)
- 4-6 years of QA/SDET experience
- Working proficiency in Python (evaluation platform, trace analysis, internal tooling are Python-based)
- Ability to analyze traces and tool calls using observability tooling (Opik, Langfuse, LangSmith, or equivalent) to determine root cause
- Strong API testing experience (majority of surface under test is API-level)
- Engineering ability to build own tooling and automation
- Clear written communication for documentation that engineering can act on directly
- Bonus: Experience with trading or exchange platforms, futures, options, crypto derivatives, margin, or settlement flows
