SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Nexxa.ai is building AI systems for heavy industries, enabling autonomous machines, systems, and operations across manufacturing, large-scale infrastructure, and logistics. The company is seeking a Senior SDET to own quality for AI agent systems that plan, call tools, and take multi-step actions autonomously in industrial environments.
You will combine deep test automation expertise with hands-on manual and exploratory testing. Your responsibilities span three core areas:
**Automation and Infrastructure:** Design and build test frameworks, evaluation harnesses, and regression suites for LLM-based agents, covering reasoning quality, tool-call correctness, task completion, and multi-turn coherence. Build automated CI/CD pipelines that run evals on every model, prompt, or tool-integration change. Develop simulators, mocks, and sandbox environments for industrial systems and tool APIs. Implement API, contract, integration, and end-to-end tests for services, tools, and orchestration layers. Develop performance, load, and reliability tests including failure injection and resilience testing.
**Manual and Exploratory Testing:** Perform hands-on exploratory testing of agent behavior, including new capabilities, ambiguous requests, long multi-turn sessions, and unusual operational scenarios. Manually review agent traces and outputs to judge reasoning quality and safety. Execute manual regression and release verification passes in staging and customer-like environments. Conduct red-teaming sessions (prompt injection, jailbreaks, tool misuse) in partnership with security teams. Support user acceptance testing and pilot rollouts with Forward Deployed Engineers. Write clear test plans, test charters, and bug reports.
**Datasets, Metrics, and Quality Strategy:** Create and manage golden datasets and labeled test sets with edge cases, ambiguous inputs, and adversarial prompts specific to industrial contexts. Design rubrics and run human-in-the-loop evaluations. Define and instrument quality metrics beyond accuracy (groundedness, hallucination rate, task success rate, latency/cost tradeoffs, safety violations) and build dashboards. Test the full agent action loop (planning, tool selection, tool execution, error recovery, output). Investigate and triage failures to identify root causes. Establish quality bars, release criteria, and sign-off processes. Mentor engineers on testing strategies for LLM-driven systems. Advocate for testability, observability, and tracing in agent architecture.
You will work closely with ML engineers, backend engineers, and Forward Deployed Engineers to define what "good" looks like for agents operating in high-stakes industrial settings, then build and run the systems and processes that measure it continuously.
**Requirements:**
- 5+ years as an SDET, QA engineer, or software engineer with strong testing focus and demonstrated ownership of test strategy and automation for complex distributed systems
- Proven experience in both manual and automated testing, including exploratory testing, test planning, and writing high-quality bug reports
- Strong programming skills (Python preferred) with software engineering mindset: clean, maintainable, well-reviewed code for frameworks, data pipelines, and tooling
- Experience building test frameworks and infrastructure from the ground up, not just writing test cases within existing frameworks
- Hands-on experience with CI/CD systems (GitHub Actions, GitLab CI, Jenkins) and containerized test environments (Docker, Kubernetes)
- Experience testing APIs and microservices, including contract, integration, and end-to-end testing
- Hands-on experience testing LLM-based products, chatbots, or AI agents, and understanding of why traditional deterministic assertions break down for generative systems
- Practical experience with eval frameworks or tooling (promptfoo, DeepEval, RAGAS, LangSmith) or track record of building your own
- Understanding of how LLM agents work: prompting, tool/function calling, context management, RAG, memory, and orchestration frameworks
- Experience designing test data and labeled datasets, including sourcing, sampling, and managing dataset drift
- Familiarity with LLM-specific failure modes: hallucination, prompt injection, context poisoning, tool misuse, goal drift, and non-determinism
- Comfort operating in ambiguity and defining what "correct" means when there's no single right answer
- Strong written communication skills for turning fuzzy quality signals into clear, actionable findings
**Preferred:**
- Experience with human-in-the-loop evaluation workflows (labeling pipelines, inter-rater reliability, rubric design)
- Experience with performance and load testing tools (k6, Locust) and chaos or fault-injection testing
- Background in ML/data science sufficient to read model evals and statistical significance
- Experience red-teaming or doing adversarial/security testing on ML systems
- Familiarity with observability and tracing tools for LLM applications (LangSmith, Arize, Langfuse, Weights & Biases) and general observability stacks (OpenTelemetry, Grafana, Datadog)
- Experience testing systems in industrial, IoT, or operational technology (OT) environments, including industrial protocols (OPC UA, MQTT, Modbus) or simulation and hardware-in-the-loop setups
- Experience supporting UAT, pilots, or customer-facing validation of complex products
- Experience setting up test and eval infrastructure from scratch at a startup or fast-moving team
