SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Salary: USD 185,000 - 210,000 / annual
Impiricus is an agentic commercialization platform for healthcare that helps life sciences teams understand physician intent, coordinate clinical and commercial resources, and enable real-time action. The company's AI-powered platform acts as a field force multiplier, extending reach between rep visits and connecting healthcare professionals to coverage, evidence, medical information, patient support, and field teams.
The Director, AI Automation Engineering is a player-coach role responsible for owning the quality and reliability layer for everything the AI organization ships. This is the first dedicated hire into a function expected to grow, and the Director will set the bar, do the hardest technical work early, and grow into leading a small automation team as coverage expands.
Key responsibilities include:
**Testing Harness and Infrastructure**: Own the infrastructure that continuously verifies whether autonomous agents are taking intended actions before and after production. Stage, test, and lifecycle-manage bots and agent workflows. Establish baseline quality metrics and coverage standards for AI initiatives. Build infrastructure to run automated research chains and agent evals at scale.
**Evaluation Frameworks**: Create automated evaluation frameworks for LLM outputs covering accuracy, safety, bias, and regression. Implement RAG evaluation, prompt regression testing, and golden-set benchmarking. Define and enforce quality thresholds that AI Engineers must meet before shipping. Partner with AI Researchers and Engineers on eval design for new model and agent patterns.
**CI/CD and Release Quality**: Build release-gating infrastructure that governs when an agent is trusted to act in production, including staged rollouts and rollback authority. Develop AI automation solutions to ensure agentic coding delivers highly functioning software while maintaining quality. Integrate testing into GitHub Actions for AI model and agent releases. Maintain existing test suites as models, prompts, and tools change.
**Cross-Team Quality Partnership**: Work with AI Solutions Architects on eval criteria for workflow automations. Partner with Principal AI Product on launch-readiness and QA for AI projects. Support product teams on AI feature validation. Document standards, runbooks, and patterns so non-engineers building in the platform can self-serve basic testing.
The role reports to the SVP, Product | AI and involves daily partnership with AI Engineers, AI Solutions Architects, and product teams shipping AI features across internal platforms (Sage) and customer-facing AI (Ascend, DocUpdate).
**Requirements**:
- 5+ years building, evaluating, or governing production AI/ML systems, with direct experience in agentic AI, LLM-based automation, or autonomous system safety
- Direct experience evaluating LLM outputs: RAG eval, prompt regression, safety and accuracy benchmarking
- Experience building CI/CD pipelines with integrated test gates (GitHub Actions or similar)
- Comfortable working across AI platform infrastructure and product-facing AI features
- Operates independently as a senior individual contributor and communicates quality standards to engineers and non-engineers
- Proficiency in Python and standard testing frameworks (pytest or equivalent)
- Strong plus: experience with agent testing, bot lifecycle management, or AI agent orchestration platforms
- Strong plus: familiarity with LLM observability and eval tooling (LangSmith, Braintrust, custom harnesses, etc.)
- Strong plus: background in healthcare, pharma, or regulated environments where quality gates matter
- Strong plus: prior experience standing up a QA or automation function from scratch, with appetite to grow into leading a team
