SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Plenful is building an AI automation platform for healthcare operations, recently funded with $50M in Series B backing. The company serves 100+ health systems, pharmacies, and healthcare organizations, automating back-office work (faxes, PDFs, phone calls) with production-grade AI agents and models that maintain accuracy for patient and financial data.
As Staff Machine Learning Engineer, you are a senior individual contributor setting technical direction across the ML organization with no direct reports but broad influence. You will:
**Agent Architecture & Strategy**: Define production agent architecture including tool use, retrieval, state/context management, routing, orchestration, tracing, and failure recovery. Establish patterns for reuse across the engineering team and make build-vs-buy decisions for models and platform components.
**Post-Training & Model Development**: Own the post-training strategy, deciding where supervised fine-tuning, preference optimization, distillation, and reinforcement learning deliver value. Train and ship LLMs and vision-language models for healthcare tasks, building in-house capability for continuous model improvement. Validate every gain with production-grade evaluation.
**AI Quality & Evaluation**: Set the AI quality strategy including automated graders, calibrated LLM judges, regression suites, human review, failure taxonomies, and offline-to-online validation. Make evaluation results the basis for release decisions.
**Self-Improving Systems**: Design systems that turn production failures and user feedback into better datasets, evals, and models. Establish standards for dataset versioning, observability, and MLOps ensuring reproducibility and measurability across teams.
**Leadership & Mentorship**: Co-own the 12-month ML roadmap with engineering and product leaders. Lead technical reviews, mentor Senior engineers, and contribute to hiring. Build privacy-by-design practices: minimum necessary data, de-identified or synthetic data where applicable, and HIPAA-aware practices throughout.
**Staying Current**: Use new models and coding agents immediately upon release. Turn papers and releases into experiments within days with measurable results. Demonstrate proof through open source, write-ups, talks, or evaluations others use.
**First 90 Days**: By day 30, review agents, evals, and training pipeline, documenting three biggest gaps. By day 90, ship a post-training or evaluation improvement that moves a production metric and publish the team's roadmap for the next two quarters.
The role is hybrid with two days per week in the San Francisco office; the company is remote-first with offices in San Francisco and New York.
**Requirements:**
- 8+ years in ML or software engineering, including 3+ years leading LLM or agentic systems in production
- Deep applied post-training experience: must have run supervised fine-tuning, preference optimization, distillation, or RL end-to-end and shipped a model that moved a production metric
- Designed an evaluation strategy for LLM or agent systems and seen it drive real release decisions
- Architected production agent systems; experience with LangGraph, RAG, and vector search (e.g., Pinecone) expected
- Strong Python and PyTorch; comfort with distributed training and serving (e.g., vLLM, FSDP, DeepSpeed)
- Real MLOps depth: Docker, Kubernetes, MLflow or equivalent, CI/CD, experiment and dataset tracking
- Ability to influence without authority; design docs, reviews, and mentoring make other engineers better
**Nice to Have:**
- Healthcare data experience: EHR, FHIR, claims, prior authorization
- Publications or open-source work in LLM evaluation, post-training, or agents
- Experience interviewing and hiring ML engineers
- Work in another regulated, high-stakes domain
