SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Ruya AI is a venture-backed defense-tech company building AI-native intelligent software for sovereign institutions in mission-critical environments. The Quant team develops a multi-modal intelligence agent that surfaces hidden patterns across disparate data sources and generates predictive outcomes for operators.
As Senior QA and AI Evals Engineer, you will own the quality system for this platform, covering both deterministic software behavior and non-deterministic AI outcomes. You'll design and build automated test suites, evaluation pipelines, and release gates that ensure reliability across user flows and probabilistic AI outputs.
Core responsibilities include:
- Build end-to-end test coverage for investigations, maps, graphs, authentication, and chat interfaces
- Develop API, contract, integration, and component tests across TypeScript and Python services
- Create Gherkin/BDD scenarios and Playwright test cases for critical UI flows
- Build AI evaluation datasets, metrics, experiments, and regression gates using Langfuse
- Test streaming disconnects, replay, cancellation, retries, race conditions, and long-running jobs
- Validate authorization boundaries and sensitive-data handling
- Integrate test suites into GitHub Actions and diagnose flaky tests
- Partner with engineers early to create testable interfaces and useful observability
Required qualifications: Strong Playwright and API/integration testing experience; pytest and TypeScript test frameworks (Vitest); understanding of asynchronous and distributed-system failure modes; experience evaluating LLM or non-deterministic systems; ability to design high-signal test suites.
Preferred: Langfuse, playwright-bdd, fault injection, contract testing, security testing, durable workflows, or model/prompt deployment gates.
Headquartered in Massachusetts with offices across the US, Europe, and Middle East. Remote arrangements available in California, Massachusetts, Texas, Maryland, Virginia, and across the US. Also hiring in Dubai, Bosnia, Croatia, and Serbia with visa sponsorship and relocation assistance available.
