SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Salary: USD 180,000 - 230,000 / annual
Ellipsis Health is building AI/ML products to solve healthcare staffing and administrative challenges using conversational AI and proprietary voice biomarker technology. As a Senior Data Scientist specializing in AI Quality & Evaluation, you will own end-to-end data science and AI evaluation projects—from problem definition through execution, analysis, and implementation.
Key responsibilities include:
• Drive high-impact projects independently, defining problems, developing methodologies, executing analyses, and communicating results to drive implementation across the organization.
• Build and maintain the data and evaluation infrastructure for a closed-loop AI quality system, including golden datasets, ground-truth benchmarks, judge calibration, error analysis, and production feedback loops. Continuously identify failure modes and translate insights into improvements to datasets, evaluation methodologies, LLM-as-a-Judge systems, and underlying AI products.
• Design and implement robust evaluation frameworks, methodologies, and metrics for LLM and speech-based AI systems. Develop scalable approaches to measure quality and accelerate customer onboarding.
• Design, curate, and maintain high-quality datasets for training, testing, benchmarking, and evaluating conversational AI systems and their components.
• Build and maintain scalable data pipelines and intuitive dashboards enabling reliable data analysis, ongoing AI quality monitoring, and data-driven decision-making across the organization.
• Collaborate closely with AI/ML Engineers and Product Managers to integrate findings into product enhancements. Stay current with advancements in AI evaluation, particularly LLMs and speech systems.
• Mentor junior data scientists, establish best practices, and raise the technical bar across the team.
Required qualifications: 5+ years in data science, machine learning, or related quantitative field. Bachelor's or Master's in Computer Science, Data Science, Statistics, Mathematics, or related field. Strong Python proficiency (Pydantic, Pandas, NumPy, Scikit-learn). Strong foundation in statistical analysis, experimental design, hypothesis testing, and A/B testing. Experience with large-scale datasets, data pipelines, and analytical tools. Familiarity with evaluation metrics (precision, recall, Cohen's Kappa). Demonstrated ability to independently own and drive projects. Excellent communication and cross-functional collaboration skills.
Bonus: NLP and LLM architecture experience; speech technology evaluation; regulated industry experience (healthcare, finance); data governance and PII/PHI knowledge; MLOps and production deployment; Langfuse, cloud platforms (GCP, AWS, Azure), or Databricks familiarity.
