SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Handshake is a career platform serving 25 million job seekers, 1 million+ employers, and 1,600 educational institutions. The company recently launched Handshake AI, a rapidly growing data business supporting frontier AI labs with human-generated data for model training and evaluation. The business has scaled from $0 to ~$1B run rate in 2025.
The Quality team builds systems that define, select, and enforce quality standards across data and evaluations delivered to leading AI labs. As a Software Engineer I on this team, you'll develop backend infrastructure that evaluates complex outputs submitted by contributors—such as coding tasks designed to challenge advanced AI models. Your work will span the entire production pipeline: providing real-time feedback to contributors, supporting internal operators, and validating outputs before customer delivery.
Key responsibilities include building and scaling backend systems for quality evaluation, developing infrastructure for real-time quality checks, designing recommendation systems to select effective quality checks per project, creating data pipelines and measurement systems for subjective quality signals, building APIs and platform capabilities for cross-team use, and surfacing Quality capabilities through MCP and AI-native interfaces. You'll partner with engineers, researchers, product teams, and operations to translate ambiguous quality problems into scalable technical solutions, and continuously improve reliability, efficiency, and cost-effectiveness as the business scales.
Required experience: 2+ years building and maintaining backend applications, services, or data-intensive systems; strong software engineering fundamentals (system design, testing, debugging, operational reliability); data-oriented problem-solving; experience with APIs, databases, data pipelines, or distributed systems; ability to navigate ambiguity and deliver iteratively; strong cross-functional communication.
Extra credit includes machine learning or applied AI experience, familiarity with model evaluation, experimentation, recommendation or ranking systems, background in annotation/data-labeling/workflow-orchestration platforms, developer platform or internal tools experience, knowledge of MCP or agentic systems, and startup environment experience.
