SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Ruya AI is a venture-backed defense-tech company building AI-native intelligent software for sovereign institutions operating in mission-critical environments. The company is headquartered in Massachusetts with offices across the US, Europe, and the Middle East.
You will join the Quant team, responsible for a multi-modal intelligence agent that surfaces hidden patterns across disparate data sources and generates predictive outcomes for operators. As Senior QA and AI Evals Engineer, you own the quality system behind the platform, covering both deterministic software behavior and non-deterministic AI outcomes.
Core Responsibilities:
- Design and build automated test suites, evaluation pipelines, and release gates ensuring reliability across deterministic user flows and probabilistic AI outputs
- Build end-to-end coverage for investigations, maps, graphs, authentication, and chat features
- Develop API, contract, integration, and component tests across TypeScript and Python services
- Create Gherkin/BDD scenarios and Playwright test cases for critical UI flows
- Build AI evaluation datasets, metrics, experiments, and regression gates using Langfuse
- Test streaming disconnects, replay, cancellation, retries, race conditions, and long-running jobs
- Validate authorization boundaries and sensitive-data handling
- Integrate test suites into GitHub Actions and diagnose flaky tests
- Partner with engineers early to create testable interfaces and useful observability
Required Qualifications:
- Strong Playwright and API/integration testing experience
- Experience with pytest and TypeScript test frameworks (Vitest)
- Understanding of asynchronous and distributed-system failure modes
- Experience evaluating LLM or other non-deterministic systems
- Ability to design high-signal test suites rather than maximizing test counts
Preferred Qualifications:
- Langfuse, playwright-bdd, fault injection, contract testing, security testing, durable workflows, or model/prompt deployment gates experience
Location: In-person role based in Belgrade, Serbia office. Visa sponsorship and relocation assistance available. The company offers competitive salary, equity through ESOP, sign-on bonus, medical/dental/vision insurance, commuter benefits, relocation assistance, and paid time off.
