SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Hippocratic AI is building Polaris, a healthcare-focused frontier AI model with specialized safety supervisors that achieves 99.95% accuracy on drug-safety benchmarks. The company is seeking a Technical Program Manager to own and scale the human-in-the-loop data labeling and evaluation engine that powers this model.
In this role, you will run end-to-end data pipelines spanning audio and conversation annotation, clinician-labeled training data, preference and RLHF data, and evaluations that gate model releases. You'll work daily with ML researchers, applied scientists, and licensed clinicians to translate their evaluation and training-signal needs into concrete labeling tasks, rubrics, and success criteria.
Key responsibilities include: managing the full lifecycle of labeling and annotation programs (scoping, task design, execution, QA, delivery); building and scaling an expert rater workforce of clinicians and specialized annotators through marketplaces, vendors, and in-house teams; authoring and iterating labeling guidelines and annotation rubrics for high-stakes clinical judgment; designing QC systems using gold sets, calibration sessions, and inter-rater agreement tracking; and driving velocity and accountability by tracking throughput, cost, and quality while managing vendor relationships and tooling decisions.
Success metrics include owning a live labeling or eval program through a major delivery milestone by Day 90, clearing critical bottlenecks, and shipping measurable quality or throughput improvements. By 12 months, you'll run multiple concurrent programs with a calibrated rater bench trusted by ML teams.
Hippocratic AI was co-founded by CEO Munjal Shah and a team of physicians, hospital leaders, and AI pioneers from institutions including Johns Hopkins, Stanford, Google, Meta, Microsoft, and NVIDIA. The company recently raised $126M in Series C funding at a $3.5B valuation, bringing total funding to $404M.
REQUIREMENTS:
- 5+ years in technical program management or data operations in an AI/ML or data-intensive environment (3+ years considered for exceptional candidates)
- 2+ years hands-on with data labeling, annotation, or model evaluation, including writing guidelines and managing quality at scale
- Experience sourcing and managing contractor or expert annotator workforces through marketplaces (e.g., Mercor, Upwork, Toloka) or labeling vendors
- Working knowledge of how annotation quality affects model training, including preference data, pairwise comparisons, and rater calibration (RLHF)
- Strong grasp of quality metrics (IAA, precision/recall on gold sets, throughput vs. accuracy tradeoffs)
- Track record of owning critical data or eval projects under time pressure, working with ML engineers and applied scientists, and driving results without direct authority
- Comfort with ambiguity in a fast-moving, healthcare-adjacent AI environment
NICE TO HAVE:
- Experience running or supporting RLHF or other training-data pipelines at an AI lab or applied ML team
- Clinical, healthcare, or other regulated, high-stakes data experience
- Familiarity with Label Studio, Scale AI, Surge AI, or SageMaker Ground Truth
- Experience qualifying specialized raters such as clinicians or bilingual SMEs
- Exposure to reward hacking and specification gaming, and how annotation design can mitigate them
