SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Salary: USD 140,000 - 180,000 / annual
HumanSignal is a human data partner for companies building AI models and products. The company specializes in real-world data creation, annotation, and delivery through its enterprise platform Label Studio and professional services team. HumanSignal works with advanced ML and AI teams to operationalize complex data collection, multimodal pipelines, and multi-step workflows.
As Head of Quality and Project Enablement, you will own two critical functions that determine success for every Data Services engagement: project enablement and quality assurance.
In project enablement, you will translate customer specification documents into production-ready materials including annotator guidelines, decision trees, edge-case libraries with worked examples, and onboarding walkthroughs. You will build qualification tests and gold-standard sets to confirm annotators understand the spec before production work begins. You will run pilot and calibration rounds at project kickoff to surface spec ambiguities early, then resolve them with the customer and delivery lead before scaling. Throughout each project, you will maintain versioned guidelines and roll out updates as new edge cases emerge, confirming the workforce has absorbed changes.
In quality pipeline design, you will design the quality plan for every project, choosing the review structure (gold tasks, overlap/consensus, multi-stage review, expert adjudication) based on task type, risk, and budget. You will set sampling methodology to hit target confidence levels, stratify by class and difficulty, and adjust review rates based on annotator performance. You will select appropriate metrics for each task (accuracy against gold, agreement measures like κ or α, per-class error rates) and set acceptance thresholds that fit task subjectivity. You will configure these workflows in the labeling platform and turn what works into reusable quality playbooks by task type.
For delivery quality control, you will own final quality sign-off—no delivery ships without meeting agreed acceptance criteria. You will monitor quality throughout each project, catch drift early, run root-cause analysis on defects, and drive corrective action for individuals and guidelines. You will produce clear quality reports for every delivery showing methodology, results, and known limitations.
Success metrics: by 90 days, a standard enablement package and quality plan template is in use on every new project with you owning sign-off on all active deliveries. By 6 months, annotators ramp to target accuracy faster, mid-project guideline churn drops, and rework and customer-reported defects are measurably down. By 12 months, quality reports and methodology are a selling point for Data Services, and playbooks are in place so the function can scale beyond you.
REQUIREMENTS:
- 6+ years in quality assurance or quality operations for data labeling, human data, or ML training/evaluation data, including experience leading a quality function or team
- Proven ability to translate complex or ambiguous specs into guidelines and training that produce consistent results
- Strong applied statistics for QA: sampling design, confidence intervals, and agreement metrics, plus ability to explain choices to customers
- Hands-on experience configuring review and QA workflows in an annotation platform
- Proficiency in Claude Code for quality analysis
- Crisp written communication (guidelines and quality reports are the product)
Nice to have: Experience with LLM, RLHF, or preference-data projects; experience with expert or domain-specialist workforces; background in instructional design; familiarity with Label Studio; experience with model-assisted QA.
