SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Vi Operate deploys AI voice agents into healthcare systems, pharma organizations, and provider networks. These agents conduct real conversations with patients, providers, and caregivers, then take action in operational systems. This is a specialized quality assurance role focused on testing and validating voice agents in production.
Unlike traditional software QA, testing voice agents presents unique challenges. Similar calls can produce different outcomes. Model updates can silently alter behavior that unit tests won't catch. When a call fails, the root cause could originate from speech recognition, the language model, voice synthesis, telephony infrastructure, or backend integrations—and transcripts alone won't reveal which. Your job is to make this testable, repeatable, and detectable before clients encounter problems.
You will own quality for agents that interact with real patients, providers, and caregivers. This includes building test and evaluation systems that engineers develop against, establishing release criteria before deployment, and diagnosing bad call recordings to pinpoint exactly what failed. The role requires expertise across classical QA, voice system testing, and LLM evaluation in production environments.
Key Responsibilities:
- Voice testing and debugging: analyze call recordings, transcripts, latency traces, and logs to isolate failures at the speech-to-text, model, text-to-speech, telephony, or integration layers
- Transform vague failure reports into reproducible, actionable tickets for engineers
- Build evaluation and regression testing frameworks: golden test sets, simulated calls, transcript judgment, accuracy validation for classification and extraction, and structured output validation
- Maintain regression coverage that remains valid when underlying models change
- Design testing strategies for non-deterministic systems: define what "passing" means when outputs vary, set tolerances, and prevent flaky tests from eroding team trust
- Establish and enforce release criteria: determine what must be true before agents go live and what must remain true post-deployment
- Conduct classical QA across the broader product: test plans, regression suites, bug triage, reproduction steps, and CI integration
- Monitor production quality: sample live calls post-release to detect drift and degradation, escalating issues before clients report them
- Red-team agents proactively: test prompt injection via documents, off-script caller behavior, social engineering attempts to extract patient information, and verify guardrails and access controls hold under pressure
Requirements:
- Proven QA experience on production voice agents: you must have tested a system that talks to people on the phone and can walk through a real failure you diagnosed, including what you heard, where you looked, and which layer was at fault (this is required and screened hardest)
- QA experience on LLM systems in production (not side projects); ability to describe how you evaluated whether the system was working
- Strong classical QA foundation: test strategy, test planning, regression suites, bug triage, and release process experience in environments where QA had organizational teeth
- Experience testing non-deterministic systems: you have made variable outputs testable and can explain the tradeoffs you made
- Python proficiency and test automation skills: you write your own tooling and integrate it into CI rather than delegating
- Sound judgment about severity: you can distinguish cosmetic defects from issues that could mislead patients on calls and defend those assessments to engineers
- Systems thinking: you examine the whole system, identify seams between components, anticipate unplanned inputs, and consider adversarial use cases
Nice to Have:
- Background in IVR, contact center, or CCaaS quality (Genesys, NICE, Five9, Amazon Connect, Twilio, or similar)
- Audio and network condition testing: codecs, jitter, packet loss, barge-in and interruption handling
- Familiarity with evaluation tooling: LLM-as-judge harnesses, prompt regression frameworks, or simulated conversation testing
- Healthcare or pharma experience and HIPAA compliance comfort
- Experience building a QA function from scratch
- Load and performance testing on real-time systems
- Spanish proficiency to listen to calls and judge agent handling
