SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Keeper Security is hiring a Senior Software Development Engineer in Test (SDET), AI & Automation to join the AI team and establish scalable quality engineering practices for AI-driven systems in security-sensitive environments. This is a 100% remote position with an option to work hybrid for candidates in El Dorado Hills, CA or Chicago, IL.
KeeperSecurity is one of the fastest-growing cybersecurity software companies, protecting thousands of organizations and millions of people in over 150 countries. The company is a pioneer of zero-knowledge and zero-trust security, with KeeperPAM® as its core AI-enabled, cloud-native platform for privileged access management.
As a Senior SDET, you will serve as an embedded quality engineering partner within Keeper's AI team, working alongside AI engineers, software engineers, and product managers during design and development to define test strategy, establish acceptance criteria, and build automated coverage for AI-powered capabilities.
This role requires expertise in testing both traditional deterministic software systems and non-deterministic AI/ML systems. Traditional application behavior can be validated against consistent expected outputs, while AI models may produce different but acceptable results across models, prompts, and inputs. You will design testing approaches that account for this variability while establishing measurable quality, reliability, and safety standards.
Key responsibilities include: partnering with AI engineers and product managers to define test strategy and automation requirements; designing and maintaining automated testing frameworks for AI-powered applications, backend services, and APIs; developing testing strategies for both deterministic and non-deterministic AI/ML outputs; defining measurable evaluation criteria and quality thresholds for AI model and LLM-driven functionality; building automated regression, integration, API, and end-to-end coverage; developing test harnesses and evaluation workflows for prompts, models, agents, and tool-calling workflows; validating AI behavior across different models, prompts, inputs, and configurations; developing automated test utilities using Python, Java, JavaScript/TypeScript, Rust, or similar languages; integrating automated validation into CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI, or similar platforms; building and maintaining test datasets, mocks, fixtures, and synthetic data; analyzing model and application failures to distinguish defects from expected variability; evaluating AI outputs for reliability, consistency, safety, and alignment with expected behavior; working with engineering teams to improve testability and observability; tracking test cases and coverage using TestRail or similar platforms; analyzing logs and telemetry to identify root causes and produce defect reports; reducing flaky tests and improving test suite stability; contributing to quality engineering standards and AI testing methodologies; and using AI-assisted tools like Claude, ChatGPT, and GitHub Copilot to improve automation development and analysis.
Requirements: 5+ years of experience in SDET, software development, test automation, or code-intensive quality engineering; hands-on experience testing AI/ML, LLM-powered, or model-integrated software systems; strong understanding of differences between deterministic software testing and non-deterministic AI/ML validation; experience defining evaluation criteria, expected behaviors, tolerances, or scoring approaches for AI-driven functionality; strong programming skills in Python, Java, JavaScript/TypeScript, Rust, or another general-purpose language; significant experience designing, developing, and maintaining automated test frameworks; experience building automated functional, integration, API, and end-to-end test coverage; experience testing REST APIs and distributed workflows using Postman, REST Assured, or similar tools; experience integrating automated tests into CI/CD pipelines; experience working within Agile teams with close QA, Engineering, and Product collaboration; strong understanding of shift-left testing, continuous quality, and testability principles; experience with TestRail or similar test management platforms; strong debugging and troubleshooting skills using logs, APIs, and telemetry; experience with Git-based workflows and modern software engineering practices; familiarity with AI-assisted development tools like Claude, ChatGPT, and GitHub Copilot; strong analytical and problem-solving skills to evaluate variable AI behavior; strong written and verbal communication skills; Bachelor's degree in Computer Science, Engineering, or related technical field, or equivalent practical experience; U.S. Person status (U.S. citizen or lawful permanent resident) required due to FedRAMP compliance.
Preferred qualifications include: experience testing LLMs, agentic systems, retrieval-augmented generation, tool calling, or prompt-driven workflows; experience building automated model evaluation, benchmarking, or regression frameworks; familiarity with AI evaluation metrics and techniques for measuring quality across non-deterministic outputs; experience with Rust in production or test automation; experience with performance, reliability, security, or adversarial testing of AI-enabled systems; experience with AI-assisted generation of automated tests or synthetic test data; familiarity with Docker, Kubernetes, or cloud-based testing environments; experience testing cybersecurity, identity, PAM, or security-sensitive software; experience building shared automation frameworks or internal testing tools.
