SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Ruya AI is a venture-backed defense-tech company building AI-native intelligent software for sovereign institutions operating in mission-critical, high-stakes environments. The company is headquartered in Massachusetts with offices across the U.S., Europe, and the Middle East.
You will join the Quant team, responsible for a multi-modal intelligence agent that surfaces hidden patterns across disparate data sources and generates predictive outcomes for operators. As Senior QA and AI Evals Engineer, you own the quality system behind the platform, covering both conventional software behavior and non-deterministic AI outcomes.
Core responsibilities include:
- Design and build end-to-end test coverage for investigations, maps, graphs, authentication, and chat features
- Develop API, contract, integration, and component tests across TypeScript and Python services
- Create Gherkin/BDD scenarios and Playwright test cases for critical UI flows
- Build AI evaluation datasets, metrics, experiments, and regression gates using Langfuse
- Test streaming disconnects, replay, cancellation, retries, race conditions, and long-running jobs
- Validate authorization boundaries and sensitive-data handling
- Integrate test suites into GitHub Actions and diagnose flaky tests
- Partner with engineers early to create testable interfaces and useful observability
Required qualifications: Strong Playwright and API/integration testing experience; expertise with pytest and TypeScript test frameworks (Vitest); understanding of asynchronous and distributed-system failure modes; experience evaluating LLM or other non-deterministic systems; ability to design high-signal test suites.
Preferred: Langfuse, playwright-bdd, fault injection, contract testing, security testing, durable workflows, or model/prompt deployment gates.
The role is based at headquarters in Massachusetts with remote arrangements available in California, Massachusetts, Texas, Maryland, Virginia, and across the US. The company also hires at locations in Dubai, Bosnia, Croatia, and Serbia. Visa sponsorship and relocation assistance available for international locations.
