SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Salary: USD 200,000 - 350,000 / annual
Hercules (hercules.app) is an AI coding agent company seeking a Staff Technical Product Manager to own the evals function. Your singular goal is to make Hercules' AI coding agent cheaper, faster, and more accurate.
You will own two critical systems: Hercules Bench (offline eval set) and Hercules online evals. You will leverage the company's 500k+ users, 1M+ code repositories, and millions of chat threads to guide eval creation while maintaining strict data anonymization to protect customer privacy.
The role demands deep technical expertise across multiple domains. You must be a strong software engineer capable of coding and understanding system design, architecture, data modeling, and API design—you'll partner with engineering counterparts on complex implementations. You need explicit experience creating and managing technical evals at scale, with deep knowledge of eval methodologies, reward modeling, and optimization techniques.
Beyond evals, you need strong product sense (understanding customer goals and building novel solutions), design sense (shipping simple, delightful UX/UI), and cutting-edge AI context and harness engineering expertise. This includes hands-on experience with summarization/compression, prompting, prompt caching, subagents, memory management, context management, tools (async, deferred), skills, and related best practices. The AI/ML landscape evolves weekly—you should have wrestled with these challenges at scale and hold strong opinions about what works.
Hercules' stack: TypeScript, React, AI SDK, Postgres, Redis, Clickhouse, Tailwind, IaC (CDK, Terraform), Docker, Otel, Hono, Drizzle ORM, Temporal. Cloud infrastructure on Cloudflare and AWS. AI models from OpenAI, Anthropic, and Google.
The company operates from San Francisco (Kearny and Bush) with a strong in-office preference. Remote work is considered for exceptional candidates willing to work US time zones and travel to the office regularly.
Operating principles: ship fast and iterate quickly; strive for excellence with good taste and attention to detail; maintain hunger and willingness to work hard and operate at any level to achieve outcomes.
Benefits include unlimited AI token budget, take-what-you-need vacation policy, healthcare/vision/dental, daily office lunches/dinners/snacks.
REQUIREMENTS:
- Explicit experience creating and/or managing technical evals
- Deep understanding of eval methodologies and reward modeling
- Strong software fundamentals: ability to code, understand system design, architecture, data modeling, API design
- Hands-on experience with AI context and harness engineering best practices (summarization, compression, prompting, prompt caching, subagents, memory, context management, tools, async tools, deferred tools, skills)
- Proficiency with core stack: TypeScript, React, agent frameworks (e.g., AI SDK), eval platforms (e.g., LangSmith)
- Strong product sense and design sense
- Ability to leverage frontier AI tools to 10x workflow
- Alignment with company operating principles: bias for action, striving for excellence, hunger to achieve outcomes
